Production Spark Environment Setup

Production Spark Environment Setup

If we use a cloud Spark cluster such as AWS EMR, there will be expenses to run our Spark code. Therefore, we need to setup our own Spark cluster instead.

Production Spark cluster will be running on Linux.

We will cover:

Docker deployment of Spark Cluster

Physical deployment of Spark Cluster