# Spark Core Introduction #### Driver program -- Application code runs on Spark At a high level, every Spark application consists of a driver program that runs the user’s main function and executes various parallel operations on a cluster. #### RDD The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. RDDs automatically recover from node failures. #### Variables A second abstraction in Spark is shared variables that can be used in parallel operations. By default, when Spark runs a function in parallel as a set of tasks on different nodes, it ships a copy of each variable used in the function to each task. Sometimes, a variable needs to be shared across tasks, or between tasks and the driver program. #### Spark supports two types of shared variables: Broadcast variables, which can be used to cache a value in memory on all nodes Accumulators, which are variables that are only “added” to, such as counters and sums. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://george-jen.gitbook.io/data-science-and-apache-spark/spark-core-introduction.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.