Setup Apache Spark

Setup Apache Spark

cd ~/

mkdir spark

cd spark

Download spark binary, in our class, we use spark 3.0.0 preview with compatibility to hadoop 2.7, which we already have hadoop 2.7 instance up and running.

wget http://apache-mirror.8birdsvideo.com/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop2.7.tgz

Unpack the tgz file

tar -xvzf spark-3.0.0-preview-bin-hadoop2.7.tgz

Create a shorter name by soft link:

ln -s spark-3.0.0-preview-bin-hadoop2.7 spark

cd spark