Run a program to estimate pi

Run a program to estimate PI, which is 3.14592…

First Scala code with Apache Spark using Eclipse IDE

Start Eclipse Scala Plug-ins, click New->Scala Project

Enter Spark, click Finish

Right click Project name spark you just created, click New->Package

Enter the name of Package: using reverse domain name, like below, click Finish

Right mouse click Package name, select New->Scala Object, enter reverse domain name+name of object which is pi in this instance, click Finish

Copy below code after the package line, in my example, “package com.jentekco.spark”

import scala.math.random
import org.apache.spark._
import org.apache.log4j._
import org.apache.spark.sql.SparkSession
object pi {
    def main(args: Array[String]): Unit = {
    Logger.getLogger("org").setLevel(Level.ERROR)
    val spark = SparkSession
      .builder
      .master("local")
      .appName("Spark Pi")
      .getOrCreate()
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.sparkContext.parallelize(1 until n, slices).map { i =>
      val x = random * 2 - 1
      val y = random * 2 - 1
      if (x*x + y*y <= 1) 1 else 0
    }.reduce(_ + _)
    println(s"Pi is roughly ${4.0 * count / (n - 1)}")
    spark.stop()
  } }

To fix it, right click project spark, choose Properties,

Choose Java Build Path->Add External JARS

Navigate to the SPARK Home folder, jars sub folder, select all jar files, then click Open

Then the errors go away, Scala code is compiled successfully.

To run the Scala pi program, click Run->Run Configuration.

Then click Java Application (Scala is Java at the end of day)

In the Name field, enter pi

in the Project field, enter Spark, in the Main class field, in my example, I enter com.jentekco.spark.pi, then click Run

See output in the Console: “Pi is roughly 3.144195720978605”

Now that you have done your development and test of the scala project, you need to make a jar file to be deployed to the production. Right mouse click on spark, select Export

Choose JAR file and click Next

Under the resource to export, select the right project, in this case, spark

Choose proper folder to export pi.jar using Browse button, click Finish when done

You can run it on your PC which already has spark home, using spark-submit:

%SPARK_HOME%\bin\spark-submit --class com.jentekco.spark.pi pi.jar

Pi is roughly 3.14279571397857