> For the complete documentation index, see [llms.txt](https://george-jen.gitbook.io/data-science-and-apache-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://george-jen.gitbook.io/data-science-and-apache-spark/table-of-contents.md).

# Contents

## Setting Up

[Computer needed for this course](/data-science-and-apache-spark/computer_needed.md)

[Spark Environment Setup](/data-science-and-apache-spark/spark_setup.md)

### [Dev environment setup, task list](/data-science-and-apache-spark/dev_setup4.md)

[JDK setup](/data-science-and-apache-spark/jdk_setup5.md)

[Download and install Anaconda Python and create virtual environment with Python 3.6](/data-science-and-apache-spark/conda_setup6.md)

[Download and install Spark](/data-science-and-apache-spark/download-and-install-spark.md)

[Scala IDE](/data-science-and-apache-spark/scala-ide.md)

[Install findspark, add spylon-kernel for scala](/data-science-and-apache-spark/install-findspark-add-spylon-kernel-for-scala.md)

[Summary](/data-science-and-apache-spark/summary2.md)

### [Production Spark Environment Setup](/data-science-and-apache-spark/production-spark-environment-setup.md)

[Docker deployment of Spark Cluster](/data-science-and-apache-spark/docker-deployment-of-spark-cluster.md)

[Create customized Apache Spark Docker container](/data-science-and-apache-spark/create-customized-apache-spark-docker-container.md)

[Dockerfile](/data-science-and-apache-spark/untitled-12.md)

[docker-compose and docker-compose.yml](/data-science-and-apache-spark/docker-compose-and-docker-compose.yml.md)

[Launch custom built Docker container with docker-compose](/data-science-and-apache-spark/launch-custom-built-docker-container-with-docker-compose.md)

[Setup Hadoop, Hive and Spark on Linux without docker](/data-science-and-apache-spark/setup-hadoop-hive-and-spark-on-linux-without-docker.md)

[Hadoop configuration](/data-science-and-apache-spark/hadoop-configuration.md)

[Hadoop setup](/data-science-and-apache-spark/hadoop-setup.md)

[Configure $HADOOP\_HOME/etc/hadoop](/data-science-and-apache-spark/configure-usdhadoop_home-etc-hadoop.md)

[HDFS](/data-science-and-apache-spark/hdfs.md)

[Start Hadoop](/data-science-and-apache-spark/start-hadoop.md)

[Work with Hadoop and HDFS file system](/data-science-and-apache-spark/work-with-hadoop-and-hdfs-file-system.md)

[Connect to Hadoop web interface port 50070](/data-science-and-apache-spark/connect-to-hadoop-web-interface-port-50070.md)

[Install Hive](/data-science-and-apache-spark/install-hive.md)

[hive home](/data-science-and-apache-spark/hive-home.md)

[Initialize hive schema](/data-science-and-apache-spark/initialize-hive-schema.md)

[Start hive metastore service](/data-science-and-apache-spark/start-hive-metastore-service..md)

[Hive client](/data-science-and-apache-spark/hive-client.md)

[Setup Apache Spark](/data-science-and-apache-spark/setup-apache-spark.md)

[Spark Home](/data-science-and-apache-spark/spark-home.md)

## Python and Scala Prep

### [Python 3 Warm Up](/data-science-and-apache-spark/python-3-warm-up.md)

[Basics](/data-science-and-apache-spark/python-basics.md)

[Iterables/Collections](/data-science-and-apache-spark/iterables-collections.md)

[Strings](/data-science-and-apache-spark/python-strings.md)

[List](/data-science-and-apache-spark/python-list.md)

[Tuple](/data-science-and-apache-spark/python-tuple.md)

[Dictionary](/data-science-and-apache-spark/python-dictionary.md)

[Set](/data-science-and-apache-spark/python-set.md)

[Conditional statement](/data-science-and-apache-spark/conditional-statement.md)

[Loop statement -- For statement](/data-science-and-apache-spark/loop-statement-for-statement.md)

[Functions and methods](/data-science-and-apache-spark/functions-and-methods.md)

[map and filter](/data-science-and-apache-spark/map-and-filter.md)

[map and filter takes function as input](/data-science-and-apache-spark/map-and-filter-takes-function-as-input.md)

[lambda](/data-science-and-apache-spark/lambda.md)

[Data structure](/data-science-and-apache-spark/data-structure.md)

[Input and if statement](/data-science-and-apache-spark/input-and-if-statement.md)

[Input from a file](/data-science-and-apache-spark/input-from-a-file.md)

[Output to a file](/data-science-and-apache-spark/output-to-a-file.md)

[Python coding exercise](/data-science-and-apache-spark/python-coding-excercise.md)

### [Scala Warm Up](/data-science-and-apache-spark/scala-warm-up.md)

[Type of Variable: Mutable or immutable](/data-science-and-apache-spark/type-of-variable-mutable-or-immutable.md)

[Scala Data Type](/data-science-and-apache-spark/scala-data-type.md)

[Array in Scala](/data-science-and-apache-spark/array-in-scala.md)

[Methods](/data-science-and-apache-spark/scala-methods.md)

[Class](/data-science-and-apache-spark/scala-class.md)

[Objects](/data-science-and-apache-spark/scala-objects.md)

[Trait](/data-science-and-apache-spark/scala-trait.md)

[Scala if statement](/data-science-and-apache-spark/scala-if-statement.md)

[Scala for loop](/data-science-and-apache-spark/scala-for-loop.md)

[Scala While Loop](/data-science-and-apache-spark/scala-while-loop.md)

[Scala Exceptions + try catch finally](/data-science-and-apache-spark/scala-exceptions-+-try-catch-finally.md)

[Scala coding exercise](/data-science-and-apache-spark/scala-coding-excercise.md)

[Run a program to estimate pi](/data-science-and-apache-spark/run-a-program-to-estimate-pi.md)

[Run Scala code with Apache Spark](/data-science-and-apache-spark/run-scala-code-with-apache-spark.md)

[Python with Apache Spark using Jupyter notebook](/data-science-and-apache-spark/python-with-apache-spark-using-jupyter-notebook.md)

## Spark Core

### [Spark Core Introduction](/data-science-and-apache-spark/spark-core-introduction.md)

[Spark and Scala Version](/data-science-and-apache-spark/spark-and-scala-version.md)

[Basic Spark Package](/data-science-and-apache-spark/basic-spark-package.md)

[Resilient Distributed Datasets (RDDs)](/data-science-and-apache-spark/resilient-distributed-datasets-rdds.md)

[RDD Operations](/data-science-and-apache-spark/rdd-operations.md)

[Passing Function to Spark](/data-science-and-apache-spark/passing-function-to-spark.md)

[Printing elements of an RDD](/data-science-and-apache-spark/printing-elements-of-an-rdd.md)

[Working with key value pair](/data-science-and-apache-spark/working-with-key-value-pair.md)

[RDD Transformation Funcitons](/data-science-and-apache-spark/rdd-transformation-funcitons.md)

[RDD Action Functions](/data-science-and-apache-spark/rdd-action-functions.md)

## Spark SQL

### [SPARK SQL Introduction](/data-science-and-apache-spark/untitled-57.md)

[SQL](/data-science-and-apache-spark/sql.md)

[datasets vs dataframe](/data-science-and-apache-spark/datasets-and-dataframes.md)

[SparkSession](/data-science-and-apache-spark/sparksession.md)

[Creating DataFrames](/data-science-and-apache-spark/creating-dataframes.md)

[Running SQL Queries Programmatically](/data-science-and-apache-spark/running-sql-queries-programmatically.md)

[Creating Datasets](/data-science-and-apache-spark/creating-datasets.md)

[Interoperating with RDD](/data-science-and-apache-spark/interoperating-with-rdd.md)

[Untyped User-Defined Aggregate Functions](/data-science-and-apache-spark/untyped-user-defined-aggregate-functions.md)

[Generic Load/Save Functions](/data-science-and-apache-spark/generic-load-save-functions.md)

[Manually specify file option](/data-science-and-apache-spark/manually-specify-file-option.md)

[Run SQL on files directly](/data-science-and-apache-spark/run-sql-on-files-directly.md)

[Save Mode](/data-science-and-apache-spark/save-mode.md)

[Saving to Persistent Tables](/data-science-and-apache-spark/saving-to-persistent-tables.md)

[Bucketing, Sorting and Partitioning](/data-science-and-apache-spark/bucketing-sorting-and-partitioning.md)

[Apache Arrow](/data-science-and-apache-spark/apache-arrow.md)

[Install Python Arrow Module PyArrow](/data-science-and-apache-spark/install-python-arrow-module-pyarrow.md)

[Issue might happen import PyArrow](/data-science-and-apache-spark/issue-might-happen-import-pyarrow.md)

[Enabling for Conversion to/from Pandas in Python](/data-science-and-apache-spark/enabling-for-conversion-to-from-pandas.md)

[Connect to any data source the same consistent way](/data-science-and-apache-spark/connect-to-any-data-source-the-same-consistent-way.md)

[Spark SQL Implementation Example in Scala](/data-science-and-apache-spark/spark-sql-implementation-example-in-scala.md)

[Run Scala code in Eclipse IDE](/data-science-and-apache-spark/run-scala-code-in-eclipse-ide.md)

[Hive Integration, run SQL or HiveQL queries on existing warehouses.](/data-science-and-apache-spark/hive-integration-run-sql-or-hiveql-queries-on-existing-warehouses..md)

[Example: Enrich JSON](/data-science-and-apache-spark/enrich-json.md)

## Spark Streaming

### [SPARK Streaming Introduction](/data-science-and-apache-spark/spark-streaming.md)

#### [Discretized Streams (DStreams)](/data-science-and-apache-spark/discretized-streams-dstreams.md)

#### [Transformations on DStreams](/data-science-and-apache-spark/transformations-on-dstreams.md)

[map(func)](/data-science-and-apache-spark/map-func.md)

[filter(func)](/data-science-and-apache-spark/filter-func.md)

[repartition(numPartitions)](/data-science-and-apache-spark/repartition-numpartitions.md)

[union(otherStream)](/data-science-and-apache-spark/union-otherstream.md)

[reduce(func)](/data-science-and-apache-spark/reduce-func.md)

[count()](/data-science-and-apache-spark/stream-count.md)

[countByValue()](/data-science-and-apache-spark/countbyvalue.md)

[reduceByKey(func, \[numTasks\])](/data-science-and-apache-spark/reducebykey-func-numtasks.md)

[join(otherStream, \[numTasks\])](/data-science-and-apache-spark/join-otherstream-numtasks.md)

[cogroup(otherStream, \[numTasks\])](/data-science-and-apache-spark/cogroup-otherstream-numtasks.md)

[transform(func)](/data-science-and-apache-spark/transform-func.md)

[updateStateByKey(func)](/data-science-and-apache-spark/updatestatebykey-func.md)

[repartition(numPartitions)](/data-science-and-apache-spark/repartition-numpartitions.md)

#### [DStream Window Operations](/data-science-and-apache-spark/dstream-window-operations.md)

#### [DStream Window Transformation](/data-science-and-apache-spark/dstream-window-transformation.md)

[countByWindow(windowLength, slideInterval)](/data-science-and-apache-spark/countbywindow-windowlength-slideinterval.md)

[reduceByWindow(func, windowLength, slideInterval)](/data-science-and-apache-spark/reducebywindow-func-windowlength-slideinterval.md)

[reduceByKeyAndWindow(func, windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/reducebykeyandwindow-func-windowlength-slideinterval-numtasks.md)

[reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/reducebykeyandwindow-func-invfunc-windowlength-slideinterval-numtasks.md)

[countByValueAndWindow(windowLength, slideInterval, \[numTasks\])](/data-science-and-apache-spark/countbyvalueandwindow-windowlength-slideinterval-numtasks.md)

[window(windowLength, slideInterval)](/data-science-and-apache-spark/window-windowlength-slideinterval.md)

[Window DStream Join Operations](broken://pages/-M1YDuovl0whC755he4G)

[Window DStream print(n)](/data-science-and-apache-spark/window-dstream-print-n.md)

[saveAsTextFiles(prefix, \[suffix\])](/data-science-and-apache-spark/untitled-92.md)

[saveAsObjectFiles(prefix, \[suffix\])](/data-science-and-apache-spark/saveasobjectfiles-prefix-suffix.md)

[saveAsHadoopFiles(prefix, \[suffix\])](/data-science-and-apache-spark/saveashadoopfiles-prefix-suffix.md)

[foreachRDD(func)](/data-science-and-apache-spark/foreachrdd-func.md)

[Spark Streaming with Twitter, you can get public tweets by using Twitter API.](/data-science-and-apache-spark/spark-streaming-with-twitter-you-can-get-public-tweets-by-using-twitter-api..md)

[Spark streaming use case with Python](/data-science-and-apache-spark/spark-streaming-use-case-with-python.md)

## Spark Graphx

### [Spark Graph Computing](/data-science-and-apache-spark/spark-graph-computing.md)

### [Spark Graph Computing Continue](/data-science-and-apache-spark/spark-graph-computing-continue.md)

### [Graphx](/data-science-and-apache-spark/graphx-1.md)

#### [Package org.apache.spark.graphx](/data-science-and-apache-spark/graphx.md)

[Edge Class](/data-science-and-apache-spark/edge-class.md)

[EdgeContext Class](/data-science-and-apache-spark/edgecontext-class.md)

[EdgeDirection Class](/data-science-and-apache-spark/edgedirection-class.md)

[EdgeRDD Class](/data-science-and-apache-spark/edgerdd-class.md)

[EdgeTriplet Class](/data-science-and-apache-spark/edgetriplet-class.md)

[Graph Class](/data-science-and-apache-spark/graph-class.md)

[GraphLoader Object](/data-science-and-apache-spark/graphloader-object.md)

[GraphOps Class](/data-science-and-apache-spark/graphops-class.md)

[GraphXUtils Object](/data-science-and-apache-spark/graphxutils-object.md)

[PartitionStrategy Trait](/data-science-and-apache-spark/partitionstrategy-trait.md)

[Pregel Object](/data-science-and-apache-spark/pregel-object.md)

[TripletFields Class](/data-science-and-apache-spark/tripletfields-class.md)

[VertexRDD Class](/data-science-and-apache-spark/vertexrdd-class.md)

#### [Package org.apache.spark.graphx.impl](/data-science-and-apache-spark/untitled.md)

[AggregatingEdgeContext Class](/data-science-and-apache-spark/aggregatingedgecontext-class.md)

[EdgeRDDImpl Class](/data-science-and-apache-spark/edgerddimpl-class.md)

[Class GraphImpl](/data-science-and-apache-spark/class-graphimpl-less-than-vd-ed-greater-than.md)

[Class VertexRDDImpl](/data-science-and-apache-spark/class-vertexrddimpl-less-than-vd-greater-than.md)

#### [Package org.apache.spark.graphx.lib](/data-science-and-apache-spark/package-org.apache.spark.graphx.lib-1.md)

[Class ConnectedComponents](/data-science-and-apache-spark/class-connectedcomponents.md)

[Class LabelPropagation](/data-science-and-apache-spark/class-labelpropagation.md)

[Class PageRank](/data-science-and-apache-spark/class-pagerank.md)

[Class ShortestPaths](/data-science-and-apache-spark/class-shortestpaths.md)

[Class StronglyConnectedComponents](/data-science-and-apache-spark/class-stronglyconnectedcomponents.md)

[Class SVDPlusPlus](/data-science-and-apache-spark/class-svdplusplus.md)

[Class SVDPlusPlus.Conf](/data-science-and-apache-spark/class-svdplusplus.conf.md)

[Class TriangleCount](/data-science-and-apache-spark/class-trianglecount.md)

#### [Package org.apache.spark.graphx.util](/data-science-and-apache-spark/package-org.apache.spark.graphx.lib.md)

[Class BytecodeUtils](/data-science-and-apache-spark/class-bytecodeutils.md)

[Class GraphGenerators](/data-science-and-apache-spark/class-graphgenerators.md)

[Graphx Example 1](/data-science-and-apache-spark/graphx-examples.md)

[Graphx Example 2](/data-science-and-apache-spark/graphx-example-2.md)

[Graphx Example 3](/data-science-and-apache-spark/graphx-example-3.md)

[Spark Graphx Describes Organization Chart Easy and Fast](/data-science-and-apache-spark/untitled-98.md)

[Page Rank with Apache Spark Graphx](/data-science-and-apache-spark/graphx-application-case-2.md)

[Bulk synchronous parallel with Google Pregel Graphx Implementation Use Cases](/data-science-and-apache-spark/graphx-application-case-3.md)

[Tree and Graph Traversal with and without Spark Graphx](/data-science-and-apache-spark/tree-and-graph-traversal.md)

## Spark Machine Learning

### [Spark Machine Learning Introduction](/data-science-and-apache-spark/spark-machine-learning.md)

#### [Binary Classification](/data-science-and-apache-spark/binary-classification.md)

#### [Multiclass Classification](/data-science-and-apache-spark/multiclass-classification.md)

#### [Regression](/data-science-and-apache-spark/regression.md)

#### [Correlation](/data-science-and-apache-spark/correlation.md)

[Image Data Source](/data-science-and-apache-spark/image-data-source.md)

[ML DataFrame is SQL DataFrame](/data-science-and-apache-spark/ml-dataframe.md)

[ML Transformer](/data-science-and-apache-spark/ml-transformer.md)

[ML Estimator](/data-science-and-apache-spark/ml-estimator.md)

[ML Pipeline](/data-science-and-apache-spark/ml-pipeline.md)

[Transformer/Estimator Parameters](/data-science-and-apache-spark/transformer-estimator-parameters.md)

[Extracting, transforming and selecting features](/data-science-and-apache-spark/extracting-transforming-and-selecting-features.md)

[TF-IDF](/data-science-and-apache-spark/tf-idf.md)

[Word2Vec](/data-science-and-apache-spark/word2vec.md)

[FeatureHasher](/data-science-and-apache-spark/featurehasher.md)

[Tokenizer](/data-science-and-apache-spark/tokenizer.md)

[CountVectorizer](/data-science-and-apache-spark/countvectorizer.md)

[StopWordRemover](/data-science-and-apache-spark/stopwordremover.md)

[n-gram](/data-science-and-apache-spark/n-gram.md)

[Binarizer](/data-science-and-apache-spark/binarizer.md)

[PCA](/data-science-and-apache-spark/pca.md)

[PolynomialExpansion](/data-science-and-apache-spark/polynomialexpansion.md)

[StringIndexer](/data-science-and-apache-spark/stringindexer.md)

[Discrete Cosine Transform (DCT)](/data-science-and-apache-spark/discrete-cosine-transform-dct.md)

[One-hot encoding](/data-science-and-apache-spark/one-hot-encoding.md)

[StandardScaler](/data-science-and-apache-spark/standardscaler.md)

[IndexToString](/data-science-and-apache-spark/indextostring.md)

[VectorIndexer](/data-science-and-apache-spark/vectorindexer.md)

[Interaction](/data-science-and-apache-spark/interaction.md)

[Normalizer](/data-science-and-apache-spark/normalizer.md)

[MinMaxScaler](/data-science-and-apache-spark/minmaxscaler.md)

[MaxAbScaler](/data-science-and-apache-spark/maxabscaler.md)

[Bucketizer](/data-science-and-apache-spark/bucketizer.md)

[ElementwiseProduct](/data-science-and-apache-spark/elementwiseproduct.md)

[SQLTransformer](/data-science-and-apache-spark/sqltransformer.md)

[VectorAssembler](/data-science-and-apache-spark/vectorassembler.md)

[VectorSizeHint](/data-science-and-apache-spark/vectorsizehint.md)

[QuantileDiscretizer](/data-science-and-apache-spark/quantilediscretizer.md)

[Imputer](/data-science-and-apache-spark/imputer.md)

[VectorSlicer](/data-science-and-apache-spark/vectorslicer.md)

[RFormula](/data-science-and-apache-spark/rformula.md)

[ChiSqSelector](/data-science-and-apache-spark/chisqselector.md)

[Locality Sensitive Hashing](/data-science-and-apache-spark/locality-sensitive-hashing.md)

[MinHash for Jaccard Distance](/data-science-and-apache-spark/minhash-for-jaccard-distance.md)

#### [Classification and Regression](/data-science-and-apache-spark/classification-and-regression.md)

[LogisticRegression](/data-science-and-apache-spark/logisticregression.md)

[OneVsRest](/data-science-and-apache-spark/onevsrest.md)

[Naive Bayes classifiers](/data-science-and-apache-spark/naive-bayes-classifiers.md)

[Decision trees](/data-science-and-apache-spark/decision-trees.md)

[Random forests](/data-science-and-apache-spark/random-forests.md)

[Gradient-boosted trees (GBTs)](/data-science-and-apache-spark/gradient-boosted-trees-gbts.md)

[Multilayer perceptron classifier](/data-science-and-apache-spark/multilayer-perceptron-classifier.md)

[Linear Support Vector Machine](/data-science-and-apache-spark/linear-support-vector-machine.md)

[Linear Regression](/data-science-and-apache-spark/linear-regression.md)

[Generalized linear regression](/data-science-and-apache-spark/generalized-linear-regression.md)

[Isotonic regression](/data-science-and-apache-spark/isotonic-regression.md)

[Decision Tree Regression](/data-science-and-apache-spark/decision-tree-regression.md)

[Random Forest Regression](/data-science-and-apache-spark/random-forest-regression.md)

[Gradient-boosted tree regression](/data-science-and-apache-spark/gradient-boosted-tree-regression.md)

[Survival regression](/data-science-and-apache-spark/survival-regression.md)

#### [Clustering](/data-science-and-apache-spark/clustering.md)

[k-means](/data-science-and-apache-spark/k-means.md)

[Latent Dirichlet allocation or LDA](/data-science-and-apache-spark/latent-dirichlet-allocation-or-lda.md)

[Bisecting k-means](/data-science-and-apache-spark/bisecting-k-means.md)

#### [A Gaussian Mixture Model](/data-science-and-apache-spark/a-gaussian-mixture-model.md)

[Collaborative filtering](/data-science-and-apache-spark/collaborative-filtering.md)

[Frequent Pattern Mining](/data-science-and-apache-spark/frequent-pattern-mining.md)

[FP-Growth](/data-science-and-apache-spark/fp-growth.md)

[PrefixSpan](/data-science-and-apache-spark/prefixspan.md)

[ML Tuning: model selection and hyperparameter tuning](/data-science-and-apache-spark/ml-tuning-model-selection-and-hyperparameter-tuning.md)

[Model selection (a.k.a. hyperparameter tuning)](/data-science-and-apache-spark/model-selection-a.k.a.-hyperparameter-tuning.md)

#### [Cross-Validation](/data-science-and-apache-spark/cross-validation.md)

[Train-Validation Split](/data-science-and-apache-spark/train-validation-split.md)

#### [Spark Machine Learning Applications](/data-science-and-apache-spark/spark-machine-learning-applications.md)

[Data Visualization with Vegas Viz and Scala with Spark ML](/data-science-and-apache-spark/data-visualization-with-vegas-viz-and-scala-with-spark-ml.md)

[Apache Spark Machine Learning with Dremio Data Lake Engine](/data-science-and-apache-spark/apache-spark-machine-learning-with-dremio-data-lake-engine.md)

[Dremio Data Lake Engine Apache Arrow Flight Connector with Spark Machine Learning](/data-science-and-apache-spark/dremio-data-lake-engine-apache-arrow-flight-connector-with-spark-machine-learning.md)

[Neural Network with Apache Spark Machine Learning Multilayer Perceptron Classifier](/data-science-and-apache-spark/neural-network-with-apache-spark-machine-learning-multilayer-perceptron-classifier.md)

## Appendix

### [Video presentation](/data-science-and-apache-spark/appendix-video-presentations.md)

## [References](/data-science-and-apache-spark/references.md)