index

Install Python findspark library to be used in standalone Python script or Jupyter notebook to run Spark application outside PySpark.

Install Jupyter notebook Spylon kernel to run Scala code inside Jupyter notebook interactively.

To install findspark library for Python, open an Anaconda command prompt.

Go to virtual environment spark you have created that has Python 3.6 and Jupyter notebook

conda activate spark

For the first time, update pip

pip install pip --upgrade

pip install findspark

Next is to install Spylon library for Jupyter notebook to run Scala commands inside Jupyter-notebook

Open an Anaconda command prompt as administrator

conda activate spark

pip install spylon-kernel

Create a kernel spec for Jupyter notebook

python -m spylon_kernel install --user