Launch custom built Docker container with docker-compose

docker-compose up --scale spark-worker=3

It is better to run as background task to free up the terminal by:

nohup docker-compose up --scale spark-worker=3 &

The above commands starts up Spark cluster docker container with 1 spark master node and 3 spark worker nodes

The worker nodes are the ones that run the Spark jobs

Launch a web browser, connect to Master node <IP address>:8080

The IP address can be the localhost or the IP address of the server that runs the docker

For all the docker containers used in the class, they are available for download

Last updated 5 years ago