Common Spark command line

$SPARK_HOME/bin/spark-submit

command to submit Spark driver program (application) to run on Spark cluster
1
$SPARK_HOME/bin/spark-submit --master spark://10.0.0.202:7077 --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.11-2.4.4.jar
2
20/04/29 16:05:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
Pi is roughly 3.1450957254786274
4
​
Copied!
​

$SPARK_HOME/bin/spark-shell

Command line interface to run Scala code with Spark. It automatically creates a SparkSession called spark and SparkContext called sc after launching. To exit spark-shell, use command :quit
1
spark-shell
2
20/04/29 15:32:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
Spark context Web UI available at http://master.hadoop.lan:4040
4
Spark context available as 'sc' (master = local[*], app id = local-1588199577814).
5
Spark session available as 'spark'.
6
Welcome to
7
____ __
8
/ __/__ ___ _____/ /__
9
_\ \/ _ \/ _ `/ __/ '_/
10
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
11
/_/
12
​
13
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
14
Type in expressions to have them evaluated.
15
Type :help for more information.
16
​
17
scala>
18
​
Copied!

$SPARK_HOME/bin/pyspark

Command line interface to run Python code with Spark, upon launching, it automatically creates SparkSession called spark and SparkContext called sc, it is Python interface, exit with quit()
1
Welcome to
2
____ __
3
/ __/__ ___ _____/ /__
4
_\ \/ _ \/ _ `/ __/ '_/
5
/__ / .__/\_,_/_/ /_/\_\ version 2.4.4
6
/_/
7
​
8
Using Python version 3.7.4 (default, Aug 13 2019 20:35:49)
9
SparkSession available as 'spark'.
10
​
Copied!

$SPARK_HOME/bin/spark-sql

A SQL client used to run SQL queries against HIVE tables through HIVE thrift server.
1
spark-sql
2
20/04/29 15:48:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
Spark master: local[*], Application Id: local-1588200525594
4
spark-sql> show schemas;
5
default
6
jentekllc
7
Time taken: 3.895 seconds, Fetched 2 row(s)
8
spark-sql> select 1;
9
1
10
Time taken: 1.802 seconds, Fetched 1 row(s)
11
spark-sql>
12
​
Copied!

$SPARK_HOME/bin/run-example

Run built in scala example in $SPARK_HOME/examples/src/main/scala/org/apache/spark/examples
1
$SPARK_HOME/bin/run-example SparkPi
2
20/04/29 15:55:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
Pi is roughly 3.1335356676783386
4
​
Copied!
​
​
​
Last modified 1yr ago
Copy link