Jupyter-notebook server

If you have a Linux VM, you can setup Jupyter-notebook server, then you can connect to the Jupyter-notebook running on the VM with a web browser and run Python and Scala code on the VM.

First, you need to use putty or ssh to the VM, you should already done below. If not, do it now from below link without skipping steps.

https://george-jen.gitbook.io/data-science-and-apache-spark/configure-virtualbox-nat-as-network-adapter-on-guest-vm-and-allow-putty-ssh-through-port-forwarding

This is to assume you already have Anaconda python 3 installed on the VM and have created virtual environment spark as outlined here. If not, follow below link to install Anaconda Python 3 and create virtual environment spark, exactly instructed without skipping any steps:

https://george-jen.gitbook.io/data-science-and-apache-spark/conda_setup6

To create Jupyter-notebook server, run below command:

jupyter notebook --generate-config

Then

vi ~/.jupyter/jupyter_notebook_config.py

search below parameter and set the value on the right side of the =, and remove # comment symbol at the beginning of the line, save and exit vi

c.NotebookApp.ip='0.0.0.0' #Permit external computer to connect
c.NotebookApp.notebook_dir = '' #Local notebook visit directory
c.NotebookApp.open_browser = False # Upon start, will not auto open a browser
c.NotebookApp.port = 8888 #Specify port
c.NotebookApp.allow_origin = '*'#Allow visit from anywhere

Then set the password of jupyter-notebook server, run below command, enter your password, and confirm password

jupyter notebook password

Then launch jupyter-notebook server by:

 nohup jupyter-notebook &

The port of jupyter-notebook server is 8888, as you have configured previously:

c.NotebookApp.port = 8888

You need to setup port forwarding on the virtualbox, in my example, make host port 33 forwarding to guest port 8888

Finally, launch web-browser to connect to the Jupyter-notebook server at port 33, first time, you will be asked to enter jupyter password you have set previously

You now have successfully setup jupyter-notebook server, and connect to jupyter-notebook remotely with a web browser.

Next task is to add a spylon kernel to this jupyter-notebook server:

https://george-jen.gitbook.io/data-science-and-apache-spark/install-findspark-add-spylon-kernel-for-scala

which in short, do below in a terminal window

conda activate spark
pip install spylon-kernel
#Create a kernel spec for Jupyter notebook
python -m spylon_kernel install --user

Now restart jupyter-notebook in the terminal, if it is already running, you need to stop it first

 nohup jupyter-notebook &

Then open a web browser to point to host machine Virtualbox port forwarding port, in our case, port 33, which forwards to port 8888 of the guest machine that Jupyter-notebook server is listening to, click the notebook "New" for the drop down list, you will see Spylon-Kernel to run Scala code interactively

for example:

http://localhost:33

To stop jupyter-notebook server

jupyter-notebook server is essentially jupyter-notebook process running in background and there is no graceful way to stop it. Only way to stop it is by Linux command "kill -9"

First find out the process id (PID) by

ps -ef | grep jupyter

Note down the <PID>s for all jupyter processes, then for each <PID>

kill -9 <PID>