Run a program to estimate pi
First Scala code with Apache Spark using Eclipse IDE
Start Eclipse Scala Plug-ins, click New->Scala Project
Enter Spark, click Finish
Right click Project name spark you just created, click New->Package
Enter the name of Package: using reverse domain name, like below, click Finish
Right mouse click Package name, select New->Scala Object, enter reverse domain name+name of object which is pi in this instance, click Finish
Copy below code after the package line, in my example, “package com.jentekco.spark”
import scala.math.random
import org.apache.spark._
import org.apache.log4j._
import org.apache.spark.sql.SparkSession
object pi {
def main(args: Array[String]): Unit = {
Logger.getLogger("org").setLevel(Level.ERROR)
val spark = SparkSession
.builder
.master("local")
.appName("Spark Pi")
.getOrCreate()
val slices = if (args.length > 0) args(0).toInt else 2
val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
val count = spark.sparkContext.parallelize(1 until n, slices).map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x*x + y*y <= 1) 1 else 0
}.reduce(_ + _)
println(s"Pi is roughly ${4.0 * count / (n - 1)}")
spark.stop()
} }
To fix it, right click project spark, choose Properties,
Choose Java Build Path->Add External JARS
Navigate to the SPARK Home folder, jars sub folder, select all jar files, then click Open
Then the errors go away, Scala code is compiled successfully.
To run the Scala pi program, click Run->Run Configuration.
Then click Java Application (Scala is Java at the end of day)
In the Name field, enter pi
in the Project field, enter Spark, in the Main class field, in my example, I enter com.jentekco.spark.pi, then click Run
See output in the Console: “Pi is roughly 3.144195720978605”
Now that you have done your development and test of the scala project, you need to make a jar file to be deployed to the production. Right mouse click on spark, select Export
Choose JAR file and click Next
Under the resource to export, select the right project, in this case, spark
Choose proper folder to export pi.jar using Browse button, click Finish when done
You can run it on your PC which already has spark home, using spark-submit:
%SPARK_HOME%\bin\spark-submit --class com.jentekco.spark.pi pi.jar
Pi is roughly 3.14279571397857
Last modified 3yr ago