> For the complete documentation index, see [llms.txt](https://george-jen.gitbook.io/data-science-and-apache-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-core-introduction.md).

# Spark Core Introduction

#### Driver program -- Application code runs on Spark

At a high level, every Spark application consists of a driver program that runs the user’s main function and executes various parallel operations on a cluster.

#### RDD

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel.

RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it.

RDDs automatically recover from node failures.

#### Variables

A second abstraction in Spark is shared variables that can be used in parallel operations. By default, when Spark runs a function in parallel as a set of tasks on different nodes, it ships a copy of each variable used in the function to each task. Sometimes, a variable needs to be shared across tasks, or between tasks and the driver program. 

#### Spark supports two types of shared variables:

Broadcast variables, which can be used to cache a value in memory on all nodes

Accumulators, which are variables that are only “added” to, such as counters and sums.