Development environment on MacOS


If there is connection refused error, you need to properly address the port connection issue by configure firewall under
system preferences -> security and privacy->firewall

Install XCode

If your MacOS is latest release, you can install XCode from App Store; otherwise, you can download and install from developer site for older version of XCode that is compatible to your MacOS

Install homebrew (or brew)

Open the Terminal, enter below
ruby -e "$(curl -fsSL"
Then follow the message on the screen

Download and install JDK 8

same process for Windows, except you need to download the dmg file for the MacOS
After install, you can find JAVA_HOME on your Mac by below command line
$(dirname $(readlink $(which javac)))/java_home
On my MacBook, it displays java home path
Append into ~/.bash_profile on my Mac
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_202.jdk/Contents/Home

Download setup Eclipse IDE with Scala plug in

Install download utility wget

brew install wget

Download and install Anaconda Python 3

Download and install Hadoop

Download and install HIVE

Download and install Spark

Install Python module findspark and Jupyter Spylon-kernel for Scala