command to submit Spark driver program (application) to run on Spark cluster
$SPARK_HOME/bin/spark-submit --master spark://10.0.0.202:7077 --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.11-2.4.4.jar
20/04/29 16:05:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pi is roughly 3.1450957254786274
$SPARK_HOME/bin/spark-shell
Command line interface to run Scala code with Spark. It automatically creates a SparkSession called spark and SparkContext called sc after launching. To exit spark-shell, use command :quit
spark-shell
20/04/29 15:32:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://master.hadoop.lan:4040
Spark context available as 'sc' (master = local[*], app id = local-1588199577814).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
$SPARK_HOME/bin/pyspark
Command line interface to run Python code with Spark, upon launching, it automatically creates SparkSession called spark and SparkContext called sc, it is Python interface, exit with quit()
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Python version 3.7.4 (default, Aug 13 2019 20:35:49)
SparkSession available as 'spark'.
$SPARK_HOME/bin/spark-sql
A SQL client used to run SQL queries against HIVE tables through HIVE thrift server.
spark-sql
20/04/29 15:48:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1588200525594
spark-sql> show schemas;
default
jentekllc
Time taken: 3.895 seconds, Fetched 2 row(s)
spark-sql> select 1;
1
Time taken: 1.802 seconds, Fetched 1 row(s)
spark-sql>
$SPARK_HOME/bin/run-example
Run built in scala example in $SPARK_HOME/examples/src/main/scala/org/apache/spark/examples
$SPARK_HOME/bin/run-example SparkPi
20/04/29 15:55:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Pi is roughly 3.1335356676783386