Hadoop setup

Create hadoop folder:

cd ~/

mkdir hadoop

cd hadoop

Download hadoop binary. In this class, we choose Hadoop 2.7.7 for compatibility with Spark, expand the tar.gz.

wget https://archive.apache.org/dist/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

tar -xvzf hadoop-2.7.7.tar.gz

cd hadoop-2.7.7

pwd

/home/bigdata2/hadoop/hadoop-2.7.7

This is the HADOOP_HOME, in my case, HADOOP_HOME=/home/bigdata2/hadoop/hadoop-2.7.7

Add below lines in ~/.bashrc file

HADOOP env variables

export HADOOP_HOME=/home/bigdata2/hadoop/hadoop-2.7.7

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Then run

source ~/.bashrc or log out and log back in