Development environment on MacOS

Pr-requisites:
If there is connection refused error, you need to properly address the port connection issue by configure firewall under 
system preferences -> security and privacy->firewall
Install XCode
If your MacOS is latest release, you can install XCode from App Store; otherwise, you can download and install from developer site for older version of XCode that is compatible to your MacOS
​https://stackoverflow.com/questions/44390162/download-install-xcode-version-without-premium-developer-account​
Install homebrew (or brew)
Open the Terminal, enter below
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Then follow the message on the screen
Download and install JDK 8
same process for Windows, except you need to download the dmg file for the MacOS
​https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html​
After install, you can find JAVA_HOME on your Mac by below command line
$(dirname $(readlink $(which javac)))/java_home
On my MacBook, it displays java home path
/Library/Java/JavaVirtualMachines/jdk1.8.0_202.jdk/Contents/Home
Append into ~/.bash_profile on my Mac
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_202.jdk/Contents/Home
Download setup Eclipse IDE with Scala plug in
​Same process as Windows install​
Install download utility wget
brew install wget
Download and install Anaconda Python 3
​Same process as windows​
Download and install Hadoop
​Same process as Linux install​
Download and install HIVE
​Same process as Linux install​
Download and install Spark
​Same process as Linux install​
Install Python module findspark and Jupyter Spylon-kernel for Scala
​Same process as Windows Install​
​
Summary
Production Spark Environment Setup
Last modified 3yr ago