# Contents

## Setting Up

[Computer needed for this course](/data-science-and-apache-spark/computer_needed.md)

[Spark Environment Setup](/data-science-and-apache-spark/spark_setup.md)

### [Dev environment setup, task list](/data-science-and-apache-spark/dev_setup4.md)

[JDK setup](/data-science-and-apache-spark/jdk_setup5.md)

[Download and install Anaconda Python and create virtual environment with Python 3.6](/data-science-and-apache-spark/conda_setup6.md)

[Download and install Spark](/data-science-and-apache-spark/download-and-install-spark.md)

[Scala IDE](/data-science-and-apache-spark/scala-ide.md)

[Install findspark, add spylon-kernel for scala](/data-science-and-apache-spark/install-findspark-add-spylon-kernel-for-scala.md)

[Summary](/data-science-and-apache-spark/summary2.md)

### [Production Spark Environment Setup](/data-science-and-apache-spark/production-spark-environment-setup.md)

[Docker deployment of Spark Cluster](/data-science-and-apache-spark/docker-deployment-of-spark-cluster.md)

[Create customized Apache Spark Docker container](/data-science-and-apache-spark/create-customized-apache-spark-docker-container.md)

[Dockerfile](/data-science-and-apache-spark/untitled-12.md)

[docker-compose and docker-compose.yml](/data-science-and-apache-spark/docker-compose-and-docker-compose.yml.md)

[Launch custom built Docker container with docker-compose](/data-science-and-apache-spark/launch-custom-built-docker-container-with-docker-compose.md)

[Setup Hadoop, Hive and Spark on Linux without docker](/data-science-and-apache-spark/setup-hadoop-hive-and-spark-on-linux-without-docker.md)

[Hadoop configuration](/data-science-and-apache-spark/hadoop-configuration.md)

[Hadoop setup](/data-science-and-apache-spark/hadoop-setup.md)

[Configure $HADOOP\_HOME/etc/hadoop](/data-science-and-apache-spark/configure-usdhadoop_home-etc-hadoop.md)

[HDFS](/data-science-and-apache-spark/hdfs.md)

[Start Hadoop](/data-science-and-apache-spark/start-hadoop.md)

[Work with Hadoop and HDFS file system](/data-science-and-apache-spark/work-with-hadoop-and-hdfs-file-system.md)

[Connect to Hadoop web interface port 50070](/data-science-and-apache-spark/connect-to-hadoop-web-interface-port-50070.md)

[Install Hive](/data-science-and-apache-spark/install-hive.md)

[hive home](/data-science-and-apache-spark/hive-home.md)

[Initialize hive schema](/data-science-and-apache-spark/initialize-hive-schema.md)

[Start hive metastore service](/data-science-and-apache-spark/start-hive-metastore-service..md)

[Hive client](/data-science-and-apache-spark/hive-client.md)

[Setup Apache Spark](/data-science-and-apache-spark/setup-apache-spark.md)

[Spark Home](/data-science-and-apache-spark/spark-home.md)

## Python and Scala Prep

### [Python 3 Warm Up](/data-science-and-apache-spark/python-3-warm-up.md)

[Basics](/data-science-and-apache-spark/python-basics.md)

[Iterables/Collections](/data-science-and-apache-spark/iterables-collections.md)

[Strings](/data-science-and-apache-spark/python-strings.md)

[List](/data-science-and-apache-spark/python-list.md)

[Tuple](/data-science-and-apache-spark/python-tuple.md)

[Dictionary](/data-science-and-apache-spark/python-dictionary.md)

[Set](/data-science-and-apache-spark/python-set.md)

[Conditional statement](/data-science-and-apache-spark/conditional-statement.md)

[Loop statement -- For statement](/data-science-and-apache-spark/loop-statement-for-statement.md)

[Functions and methods](/data-science-and-apache-spark/functions-and-methods.md)

[map and filter](/data-science-and-apache-spark/map-and-filter.md)

[map and filter takes function as input](/data-science-and-apache-spark/map-and-filter-takes-function-as-input.md)

[lambda](/data-science-and-apache-spark/lambda.md)

[Data structure](/data-science-and-apache-spark/data-structure.md)

[Input and if statement](/data-science-and-apache-spark/input-and-if-statement.md)

[Input from a file](/data-science-and-apache-spark/input-from-a-file.md)

[Output to a file](/data-science-and-apache-spark/output-to-a-file.md)

[Python coding exercise](/data-science-and-apache-spark/python-coding-excercise.md)

### [Scala Warm Up](/data-science-and-apache-spark/scala-warm-up.md)

[Type of Variable: Mutable or immutable](/data-science-and-apache-spark/type-of-variable-mutable-or-immutable.md)

[Scala Data Type](/data-science-and-apache-spark/scala-data-type.md)

[Array in Scala](/data-science-and-apache-spark/array-in-scala.md)

[Methods](/data-science-and-apache-spark/scala-methods.md)

[Class](/data-science-and-apache-spark/scala-class.md)

[Objects](/data-science-and-apache-spark/scala-objects.md)

[Trait](/data-science-and-apache-spark/scala-trait.md)

[Scala if statement](/data-science-and-apache-spark/scala-if-statement.md)

[Scala for loop](/data-science-and-apache-spark/scala-for-loop.md)

[Scala While Loop](/data-science-and-apache-spark/scala-while-loop.md)

[Scala Exceptions + try catch finally](/data-science-and-apache-spark/scala-exceptions-+-try-catch-finally.md)

[Scala coding exercise](/data-science-and-apache-spark/scala-coding-excercise.md)

[Run a program to estimate pi](/data-science-and-apache-spark/run-a-program-to-estimate-pi.md)

[Run Scala code with Apache Spark](/data-science-and-apache-spark/run-scala-code-with-apache-spark.md)

[Python with Apache Spark using Jupyter notebook](/data-science-and-apache-spark/python-with-apache-spark-using-jupyter-notebook.md)

## Spark Core

### [Spark Core Introduction](/data-science-and-apache-spark/spark-core-introduction.md)

[Spark and Scala Version](/data-science-and-apache-spark/spark-and-scala-version.md)

[Basic Spark Package](/data-science-and-apache-spark/basic-spark-package.md)

[Resilient Distributed Datasets (RDDs)](/data-science-and-apache-spark/resilient-distributed-datasets-rdds.md)

[RDD Operations](/data-science-and-apache-spark/rdd-operations.md)

[Passing Function to Spark](/data-science-and-apache-spark/passing-function-to-spark.md)

[Printing elements of an RDD](/data-science-and-apache-spark/printing-elements-of-an-rdd.md)

[Working with key value pair](/data-science-and-apache-spark/working-with-key-value-pair.md)

[RDD Transformation Funcitons](/data-science-and-apache-spark/rdd-transformation-funcitons.md)

[RDD Action Functions](/data-science-and-apache-spark/rdd-action-functions.md)

## Spark SQL

### [SPARK SQL Introduction](/data-science-and-apache-spark/untitled-57.md)

[SQL](/data-science-and-apache-spark/sql.md)

[datasets vs dataframe](/data-science-and-apache-spark/datasets-and-dataframes.md)

[SparkSession](/data-science-and-apache-spark/sparksession.md)

[Creating DataFrames](/data-science-and-apache-spark/creating-dataframes.md)

[Running SQL Queries Programmatically](/data-science-and-apache-spark/running-sql-queries-programmatically.md)

[Creating Datasets](/data-science-and-apache-spark/creating-datasets.md)

[Interoperating with RDD](/data-science-and-apache-spark/interoperating-with-rdd.md)

[Untyped User-Defined Aggregate Functions](/data-science-and-apache-spark/untyped-user-defined-aggregate-functions.md)

[Generic Load/Save Functions](/data-science-and-apache-spark/generic-load-save-functions.md)

[Manually specify file option](/data-science-and-apache-spark/manually-specify-file-option.md)

[Run SQL on files directly](/data-science-and-apache-spark/run-sql-on-files-directly.md)

[Save Mode](/data-science-and-apache-spark/save-mode.md)

[Saving to Persistent Tables](/data-science-and-apache-spark/saving-to-persistent-tables.md)

[Bucketing, Sorting and Partitioning](/data-science-and-apache-spark/bucketing-sorting-and-partitioning.md)

[Apache Arrow](/data-science-and-apache-spark/apache-arrow.md)

[Install Python Arrow Module PyArrow](/data-science-and-apache-spark/install-python-arrow-module-pyarrow.md)

[Issue might happen import PyArrow](/data-science-and-apache-spark/issue-might-happen-import-pyarrow.md)

[Enabling for Conversion to/from Pandas in Python](/data-science-and-apache-spark/enabling-for-conversion-to-from-pandas.md)

[Connect to any data source the same consistent way](/data-science-and-apache-spark/connect-to-any-data-source-the-same-consistent-way.md)

[Spark SQL Implementation Example in Scala](/data-science-and-apache-spark/spark-sql-implementation-example-in-scala.md)

[Run Scala code in Eclipse IDE](/data-science-and-apache-spark/run-scala-code-in-eclipse-ide.md)

[Hive Integration, run SQL or HiveQL queries on existing warehouses.](/data-science-and-apache-spark/hive-integration-run-sql-or-hiveql-queries-on-existing-warehouses..md)

[Example: Enrich JSON](/data-science-and-apache-spark/enrich-json.md)

## Spark Streaming

### [SPARK Streaming Introduction](/data-science-and-apache-spark/spark-streaming.md)

#### [Discretized Streams (DStreams)](/data-science-and-apache-spark/discretized-streams-dstreams.md)

#### [Transformations on DStreams](/data-science-and-apache-spark/transformations-on-dstreams.md)

[map(func)](/data-science-and-apache-spark/map-func.md)

[filter(func)](/data-science-and-apache-spark/filter-func.md)

[repartition(numPartitions)](/data-science-and-apache-spark/repartition-numpartitions.md)

[union(otherStream)](/data-science-and-apache-spark/union-otherstream.md)

[reduce(func)](/data-science-and-apache-spark/reduce-func.md)

[count()](/data-science-and-apache-spark/stream-count.md)

[countByValue()](/data-science-and-apache-spark/countbyvalue.md)

[reduceByKey(func, \[numTasks\])](/data-science-and-apache-spark/reducebykey-func-numtasks.md)

[join(otherStream, \[numTasks\])](/data-science-and-apache-spark/join-otherstream-numtasks.md)

[cogroup(otherStream, \[numTasks\])](/data-science-and-apache-spark/cogroup-otherstream-numtasks.md)

[transform(func)](/data-science-and-apache-spark/transform-func.md)

[updateStateByKey(func)](/data-science-and-apache-spark/updatestatebykey-func.md)

[repartition(numPartitions)](/data-science-and-apache-spark/repartition-numpartitions.md)

#### [DStream Window Operations](/data-science-and-apache-spark/dstream-window-operations.md)

#### [DStream Window Transformation](/data-science-and-apache-spark/dstream-window-transformation.md)

[countByWindow(windowLength, slideInterval)](/data-science-and-apache-spark/countbywindow-windowlength-slideinterval.md)

[reduceByWindow(func, windowLength, slideInterval)](/data-science-and-apache-spark/reducebywindow-func-windowlength-slideinterval.md)

[reduceByKeyAndWindow(func, windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/reducebykeyandwindow-func-windowlength-slideinterval-numtasks.md)

[reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/reducebykeyandwindow-func-invfunc-windowlength-slideinterval-numtasks.md)

[countByValueAndWindow(windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/countbyvalueandwindow-windowlength-slideinterval-numtasks.md)

[window(windowLength, slideInterval)](/data-science-and-apache-spark/window-windowlength-slideinterval.md)

[Window DStream Join Operations](broken://pages/-M1YDuovl0whC755he4G)

[Window DStream print(n)](/data-science-and-apache-spark/window-dstream-print-n.md)

[saveAsTextFiles(prefix, \[suffix\])](/data-science-and-apache-spark/untitled-92.md)

[saveAsObjectFiles(prefix, \[suffix\])](/data-science-and-apache-spark/saveasobjectfiles-prefix-suffix.md)

[saveAsHadoopFiles(prefix, \[suffix\])](/data-science-and-apache-spark/saveashadoopfiles-prefix-suffix.md)

[foreachRDD(func)](/data-science-and-apache-spark/foreachrdd-func.md)

[Spark Streaming with Twitter, you can get public tweets by using Twitter API.](/data-science-and-apache-spark/spark-streaming-with-twitter-you-can-get-public-tweets-by-using-twitter-api..md)

[Spark streaming use case with Python](/data-science-and-apache-spark/spark-streaming-use-case-with-python.md)

## Spark Graphx

### [Spark Graph Computing](/data-science-and-apache-spark/spark-graph-computing.md)

### [Spark Graph Computing Continue](/data-science-and-apache-spark/spark-graph-computing-continue.md)

### [Graphx](/data-science-and-apache-spark/graphx-1.md)

#### [Package org.apache.spark.graphx](/data-science-and-apache-spark/graphx.md)

[Edge Class](/data-science-and-apache-spark/edge-class.md)

[EdgeContext Class](/data-science-and-apache-spark/edgecontext-class.md)

[EdgeDirection Class](/data-science-and-apache-spark/edgedirection-class.md)

[EdgeRDD Class](/data-science-and-apache-spark/edgerdd-class.md)

[EdgeTriplet Class](/data-science-and-apache-spark/edgetriplet-class.md)

[Graph Class](/data-science-and-apache-spark/graph-class.md)

[GraphLoader Object](/data-science-and-apache-spark/graphloader-object.md)

[GraphOps Class](/data-science-and-apache-spark/graphops-class.md)

[GraphXUtils Object](/data-science-and-apache-spark/graphxutils-object.md)

[PartitionStrategy Trait](/data-science-and-apache-spark/partitionstrategy-trait.md)

[Pregel Object](/data-science-and-apache-spark/pregel-object.md)

[TripletFields Class](/data-science-and-apache-spark/tripletfields-class.md)

[VertexRDD Class](/data-science-and-apache-spark/vertexrdd-class.md)

#### [Package org.apache.spark.graphx.impl](/data-science-and-apache-spark/untitled.md)

[AggregatingEdgeContext Class](/data-science-and-apache-spark/aggregatingedgecontext-class.md)

[EdgeRDDImpl Class](/data-science-and-apache-spark/edgerddimpl-class.md)

[Class GraphImpl](/data-science-and-apache-spark/class-graphimpl-less-than-vd-ed-greater-than.md)

[Class VertexRDDImpl](/data-science-and-apache-spark/class-vertexrddimpl-less-than-vd-greater-than.md)

#### [Package org.apache.spark.graphx.lib](/data-science-and-apache-spark/package-org.apache.spark.graphx.lib-1.md)

[Class ConnectedComponents](/data-science-and-apache-spark/class-connectedcomponents.md)

[Class LabelPropagation](/data-science-and-apache-spark/class-labelpropagation.md)

[Class PageRank](/data-science-and-apache-spark/class-pagerank.md)

[Class ShortestPaths](/data-science-and-apache-spark/class-shortestpaths.md)

[Class StronglyConnectedComponents](/data-science-and-apache-spark/class-stronglyconnectedcomponents.md)

[Class SVDPlusPlus](/data-science-and-apache-spark/class-svdplusplus.md)

[Class SVDPlusPlus.Conf](/data-science-and-apache-spark/class-svdplusplus.conf.md)

[Class TriangleCount](/data-science-and-apache-spark/class-trianglecount.md)

#### [Package org.apache.spark.graphx.util](/data-science-and-apache-spark/package-org.apache.spark.graphx.lib.md)

[Class BytecodeUtils](/data-science-and-apache-spark/class-bytecodeutils.md)

[Class GraphGenerators](/data-science-and-apache-spark/class-graphgenerators.md)

[Graphx Example 1](/data-science-and-apache-spark/graphx-examples.md)

[Graphx Example 2](/data-science-and-apache-spark/graphx-example-2.md)

[Graphx Example 3](/data-science-and-apache-spark/graphx-example-3.md)

[Spark Graphx Describes Organization Chart Easy and Fast](/data-science-and-apache-spark/untitled-98.md)

[Page Rank with Apache Spark Graphx](/data-science-and-apache-spark/graphx-application-case-2.md)

[Bulk synchronous parallel with Google Pregel Graphx Implementation Use Cases](/data-science-and-apache-spark/graphx-application-case-3.md)

[Tree and Graph Traversal with and without Spark Graphx](/data-science-and-apache-spark/tree-and-graph-traversal.md)

## Spark Machine Learning

### [Spark Machine Learning Introduction](/data-science-and-apache-spark/spark-machine-learning.md)

#### [Binary Classification](/data-science-and-apache-spark/binary-classification.md)

#### [Multiclass Classification](/data-science-and-apache-spark/multiclass-classification.md)

#### [Regression](/data-science-and-apache-spark/regression.md)

#### [Correlation](/data-science-and-apache-spark/correlation.md)

[Image Data Source](/data-science-and-apache-spark/image-data-source.md)

[ML DataFrame is SQL DataFrame](/data-science-and-apache-spark/ml-dataframe.md)

[ML Transformer](/data-science-and-apache-spark/ml-transformer.md)

[ML Estimator](/data-science-and-apache-spark/ml-estimator.md)

[ML Pipeline](/data-science-and-apache-spark/ml-pipeline.md)

[Transformer/Estimator Parameters](/data-science-and-apache-spark/transformer-estimator-parameters.md)

[Extracting, transforming and selecting features](/data-science-and-apache-spark/extracting-transforming-and-selecting-features.md)

[TF-IDF](/data-science-and-apache-spark/tf-idf.md)

[Word2Vec](/data-science-and-apache-spark/word2vec.md)

[FeatureHasher](/data-science-and-apache-spark/featurehasher.md)

[Tokenizer](/data-science-and-apache-spark/tokenizer.md)

[CountVectorizer](/data-science-and-apache-spark/countvectorizer.md)

[StopWordRemover](/data-science-and-apache-spark/stopwordremover.md)

[n-gram](/data-science-and-apache-spark/n-gram.md)

[Binarizer](/data-science-and-apache-spark/binarizer.md)

[PCA](/data-science-and-apache-spark/pca.md)

[PolynomialExpansion](/data-science-and-apache-spark/polynomialexpansion.md)

[StringIndexer](/data-science-and-apache-spark/stringindexer.md)

[Discrete Cosine Transform (DCT)](/data-science-and-apache-spark/discrete-cosine-transform-dct.md)

[One-hot encoding](/data-science-and-apache-spark/one-hot-encoding.md)

[StandardScaler](/data-science-and-apache-spark/standardscaler.md)

[IndexToString](/data-science-and-apache-spark/indextostring.md)

[VectorIndexer](/data-science-and-apache-spark/vectorindexer.md)

[Interaction](/data-science-and-apache-spark/interaction.md)

[Normalizer](/data-science-and-apache-spark/normalizer.md)

[MinMaxScaler](/data-science-and-apache-spark/minmaxscaler.md)

[MaxAbScaler](/data-science-and-apache-spark/maxabscaler.md)

[Bucketizer](/data-science-and-apache-spark/bucketizer.md)

[ElementwiseProduct](/data-science-and-apache-spark/elementwiseproduct.md)

[SQLTransformer](/data-science-and-apache-spark/sqltransformer.md)

[VectorAssembler](/data-science-and-apache-spark/vectorassembler.md)

[VectorSizeHint](/data-science-and-apache-spark/vectorsizehint.md)

[QuantileDiscretizer](/data-science-and-apache-spark/quantilediscretizer.md)

[Imputer](/data-science-and-apache-spark/imputer.md)

[VectorSlicer](/data-science-and-apache-spark/vectorslicer.md)

[RFormula](/data-science-and-apache-spark/rformula.md)

[ChiSqSelector](/data-science-and-apache-spark/chisqselector.md)

[Locality Sensitive Hashing](/data-science-and-apache-spark/locality-sensitive-hashing.md)

[MinHash for Jaccard Distance](/data-science-and-apache-spark/minhash-for-jaccard-distance.md)

#### [Classification and Regression](/data-science-and-apache-spark/classification-and-regression.md)

[LogisticRegression](/data-science-and-apache-spark/logisticregression.md)

[OneVsRest](/data-science-and-apache-spark/onevsrest.md)

[Naive Bayes classifiers](/data-science-and-apache-spark/naive-bayes-classifiers.md)

[Decision trees](/data-science-and-apache-spark/decision-trees.md)

[Random forests](/data-science-and-apache-spark/random-forests.md)

[Gradient-boosted trees (GBTs)](/data-science-and-apache-spark/gradient-boosted-trees-gbts.md)

[Multilayer perceptron classifier](/data-science-and-apache-spark/multilayer-perceptron-classifier.md)

[Linear Support Vector Machine](/data-science-and-apache-spark/linear-support-vector-machine.md)

[Linear Regression](/data-science-and-apache-spark/linear-regression.md)

[Generalized linear regression](/data-science-and-apache-spark/generalized-linear-regression.md)

[Isotonic regression](/data-science-and-apache-spark/isotonic-regression.md)

[Decision Tree Regression](/data-science-and-apache-spark/decision-tree-regression.md)

[Random Forest Regression](/data-science-and-apache-spark/random-forest-regression.md)

[Gradient-boosted tree regression](/data-science-and-apache-spark/gradient-boosted-tree-regression.md)

[Survival regression](/data-science-and-apache-spark/survival-regression.md)

#### [Clustering](/data-science-and-apache-spark/clustering.md)

[k-means](/data-science-and-apache-spark/k-means.md)

[Latent Dirichlet allocation or LDA](/data-science-and-apache-spark/latent-dirichlet-allocation-or-lda.md)

[Bisecting k-means](/data-science-and-apache-spark/bisecting-k-means.md)

#### [A Gaussian Mixture Model](/data-science-and-apache-spark/a-gaussian-mixture-model.md)

[Collaborative filtering](/data-science-and-apache-spark/collaborative-filtering.md)

[Frequent Pattern Mining](/data-science-and-apache-spark/frequent-pattern-mining.md)

[FP-Growth](/data-science-and-apache-spark/fp-growth.md)

[PrefixSpan](/data-science-and-apache-spark/prefixspan.md)

[ML Tuning: model selection and hyperparameter tuning](/data-science-and-apache-spark/ml-tuning-model-selection-and-hyperparameter-tuning.md)

[Model selection (a.k.a. hyperparameter tuning)](/data-science-and-apache-spark/model-selection-a.k.a.-hyperparameter-tuning.md)

#### [Cross-Validation](/data-science-and-apache-spark/cross-validation.md)

[Train-Validation Split](/data-science-and-apache-spark/train-validation-split.md)

#### [Spark Machine Learning Applications](/data-science-and-apache-spark/spark-machine-learning-applications.md)

[Data Visualization with Vegas Viz and Scala with Spark ML](/data-science-and-apache-spark/data-visualization-with-vegas-viz-and-scala-with-spark-ml.md)

[Apache Spark Machine Learning with Dremio Data Lake Engine](/data-science-and-apache-spark/apache-spark-machine-learning-with-dremio-data-lake-engine.md)

[Dremio Data Lake Engine Apache Arrow Flight Connector with Spark Machine Learning](/data-science-and-apache-spark/dremio-data-lake-engine-apache-arrow-flight-connector-with-spark-machine-learning.md)

[Neural Network with Apache Spark Machine Learning Multilayer Perceptron Classifier](/data-science-and-apache-spark/neural-network-with-apache-spark-machine-learning-multilayer-perceptron-classifier.md)

## Appendix

### [Video presentation](/data-science-and-apache-spark/appendix-video-presentations.md)

## [References](/data-science-and-apache-spark/references.md)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://george-jen.gitbook.io/data-science-and-apache-spark/table-of-contents.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.