# Data Science with Apache Spark ## Data Science with Apache Spark - [Preface](https://george-jen.gitbook.io/data-science-and-apache-spark/master.md) - [Contents](https://george-jen.gitbook.io/data-science-and-apache-spark/table-of-contents.md): all you need is to just click the links... - [Basic Prerequisite Skills](https://george-jen.gitbook.io/data-science-and-apache-spark/basic-prerequisite-skills.md) - [Computer needed for this course](https://george-jen.gitbook.io/data-science-and-apache-spark/computer_needed.md) - [Spark Environment Setup](https://george-jen.gitbook.io/data-science-and-apache-spark/spark_setup.md) - [Dev environment setup, task list](https://george-jen.gitbook.io/data-science-and-apache-spark/dev_setup4.md) - [JDK setup](https://george-jen.gitbook.io/data-science-and-apache-spark/jdk_setup5.md) - [Download and install Anaconda Python and create virtual environment with Python 3.6](https://george-jen.gitbook.io/data-science-and-apache-spark/conda_setup6.md) - [Download and install Spark](https://george-jen.gitbook.io/data-science-and-apache-spark/download-and-install-spark.md) - [Eclipse, the Scala IDE](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-ide.md) - [Install findspark, add spylon-kernel for scala](https://george-jen.gitbook.io/data-science-and-apache-spark/install-findspark-add-spylon-kernel-for-scala.md) - [ssh and scp client](https://george-jen.gitbook.io/data-science-and-apache-spark/ssh-and-scp-client.md) - [Summary](https://george-jen.gitbook.io/data-science-and-apache-spark/summary2.md) - [Development environment on MacOS](https://george-jen.gitbook.io/data-science-and-apache-spark/development-environment-on-macos.md) - [Production Spark Environment Setup](https://george-jen.gitbook.io/data-science-and-apache-spark/production-spark-environment-setup.md) - [VirtualBox VM](https://george-jen.gitbook.io/data-science-and-apache-spark/virtualbox-vm.md) - [VirtualBox only shows 32bit on AMD CPU](https://george-jen.gitbook.io/data-science-and-apache-spark/virtualbox-only-shows-32bit-on-amd-cpu.md) - [Configure VirtualBox NAT as Network Adapter on Guest VM and Allow putty ssh Through Port Forwarding](https://george-jen.gitbook.io/data-science-and-apache-spark/configure-virtualbox-nat-as-network-adapter-on-guest-vm-and-allow-putty-ssh-through-port-forwarding.md) - [Docker deployment of Spark Cluster](https://george-jen.gitbook.io/data-science-and-apache-spark/docker-deployment-of-spark-cluster.md) - [Create customized Apache Spark Docker container](https://george-jen.gitbook.io/data-science-and-apache-spark/create-customized-apache-spark-docker-container.md) - [Dockerfile](https://george-jen.gitbook.io/data-science-and-apache-spark/untitled-12.md) - [docker-compose and docker-compose.yml](https://george-jen.gitbook.io/data-science-and-apache-spark/docker-compose-and-docker-compose.yml.md) - [Launch custom built Docker container with docker-compose](https://george-jen.gitbook.io/data-science-and-apache-spark/launch-custom-built-docker-container-with-docker-compose.md) - [Entering Docker Container](https://george-jen.gitbook.io/data-science-and-apache-spark/entering-docker-container.md) - [Setup Hadoop, Hive and Spark on Linux without docker](https://george-jen.gitbook.io/data-science-and-apache-spark/setup-hadoop-hive-and-spark-on-linux-without-docker.md) - [Hadoop Preparation](https://george-jen.gitbook.io/data-science-and-apache-spark/hadoop-configuration.md) - [Hadoop setup](https://george-jen.gitbook.io/data-science-and-apache-spark/hadoop-setup.md) - [Configure $HADOOP\_HOME/etc/hadoop](https://george-jen.gitbook.io/data-science-and-apache-spark/configure-usdhadoop_home-etc-hadoop.md) - [HDFS](https://george-jen.gitbook.io/data-science-and-apache-spark/hdfs.md) - [Start and stop Hadoop](https://george-jen.gitbook.io/data-science-and-apache-spark/start-hadoop.md) - [Work with Hadoop and HDFS file system](https://george-jen.gitbook.io/data-science-and-apache-spark/work-with-hadoop-and-hdfs-file-system.md) - [Connect to Hadoop web interface port 50070 and 8088](https://george-jen.gitbook.io/data-science-and-apache-spark/connect-to-hadoop-web-interface-port-50070.md) - [Install Hive](https://george-jen.gitbook.io/data-science-and-apache-spark/install-hive.md) - [hive home](https://george-jen.gitbook.io/data-science-and-apache-spark/hive-home.md) - [Initialize hive schema](https://george-jen.gitbook.io/data-science-and-apache-spark/initialize-hive-schema.md) - [Start hive metastore service.](https://george-jen.gitbook.io/data-science-and-apache-spark/start-hive-metastore-service..md) - [hive-site.xml](https://george-jen.gitbook.io/data-science-and-apache-spark/hive-site.xml.md) - [Hive client](https://george-jen.gitbook.io/data-science-and-apache-spark/hive-client.md) - [Setup Apache Spark](https://george-jen.gitbook.io/data-science-and-apache-spark/setup-apache-spark.md) - [Spark Home](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-home.md) - [Jupyter-notebook server](https://george-jen.gitbook.io/data-science-and-apache-spark/jupyter-notebook-server.md) - [Python 3 Warm Up](https://george-jen.gitbook.io/data-science-and-apache-spark/python-3-warm-up.md) - [Basics](https://george-jen.gitbook.io/data-science-and-apache-spark/python-basics.md) - [Iterables/Collections](https://george-jen.gitbook.io/data-science-and-apache-spark/iterables-collections.md) - [Strings](https://george-jen.gitbook.io/data-science-and-apache-spark/python-strings.md) - [List](https://george-jen.gitbook.io/data-science-and-apache-spark/python-list.md) - [Tuple](https://george-jen.gitbook.io/data-science-and-apache-spark/python-tuple.md) - [Dictionary](https://george-jen.gitbook.io/data-science-and-apache-spark/python-dictionary.md) - [Set](https://george-jen.gitbook.io/data-science-and-apache-spark/python-set.md) - [Conditional statement](https://george-jen.gitbook.io/data-science-and-apache-spark/conditional-statement.md) - [for loop](https://george-jen.gitbook.io/data-science-and-apache-spark/loop-statement-for-statement.md) - [while loop](https://george-jen.gitbook.io/data-science-and-apache-spark/while-loop.md) - [Functions and methods](https://george-jen.gitbook.io/data-science-and-apache-spark/functions-and-methods.md) - [map and filter](https://george-jen.gitbook.io/data-science-and-apache-spark/map-and-filter.md) - [map and filter takes function as input](https://george-jen.gitbook.io/data-science-and-apache-spark/map-and-filter-takes-function-as-input.md) - [lambda](https://george-jen.gitbook.io/data-science-and-apache-spark/lambda.md) - [Python Class](https://george-jen.gitbook.io/data-science-and-apache-spark/data-structure.md) - [Input and if statement](https://george-jen.gitbook.io/data-science-and-apache-spark/input-and-if-statement.md) - [Input from a file](https://george-jen.gitbook.io/data-science-and-apache-spark/input-from-a-file.md) - [Output to a file](https://george-jen.gitbook.io/data-science-and-apache-spark/output-to-a-file.md) - [try except](https://george-jen.gitbook.io/data-science-and-apache-spark/try-except.md) - [Python coding exercise](https://george-jen.gitbook.io/data-science-and-apache-spark/python-coding-excercise.md) - [Scala Warm Up](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-warm-up.md) - [Start Spylon-kernel on Jupyter-notebook](https://george-jen.gitbook.io/data-science-and-apache-spark/start-spylon-kernel-on-jupyter-notebook.md) - [Type of Variable: Mutable or immutable](https://george-jen.gitbook.io/data-science-and-apache-spark/type-of-variable-mutable-or-immutable.md) - [Block statement](https://george-jen.gitbook.io/data-science-and-apache-spark/block-statement.md) - [Scala Data Type](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-data-type.md) - [Array in Scala](https://george-jen.gitbook.io/data-science-and-apache-spark/array-in-scala.md) - [Methods](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-methods.md) - [Functions](https://george-jen.gitbook.io/data-science-and-apache-spark/functions.md) - [Anonymous function](https://george-jen.gitbook.io/data-science-and-apache-spark/anonymous-function.md) - [Scala map and filter methods](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-map-and-filter-methods.md) - [Class](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-class.md) - [Objects](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-objects.md) - [Trait](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-trait.md) - [Tuple in Scala](https://george-jen.gitbook.io/data-science-and-apache-spark/tuple-in-scala.md) - [List/Seq](https://george-jen.gitbook.io/data-science-and-apache-spark/list-seq.md) - [Set in Scala](https://george-jen.gitbook.io/data-science-and-apache-spark/set-in-scala.md) - [Scala Map](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-map.md) - [Scala if statement](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-if-statement.md) - [Scala for loop](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-for-loop.md) - [Scala While Loop](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-while-loop.md) - [Scala Exceptions + try catch finally](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-exceptions-+-try-catch-finally.md) - [Scala coding exercise](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-coding-excercise.md) - [Run a program to estimate pi](https://george-jen.gitbook.io/data-science-and-apache-spark/run-a-program-to-estimate-pi.md) - [Common Spark command line](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-command-line.md) - [Run Scala code with spark-submit](https://george-jen.gitbook.io/data-science-and-apache-spark/run-scala-code-with-apache-spark.md) - [Python with Apache Spark using Jupyter notebook](https://george-jen.gitbook.io/data-science-and-apache-spark/python-with-apache-spark-using-jupyter-notebook.md) - [Spark Core Introduction](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-core-introduction.md) - [Spark and Scala Version](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-and-scala-version.md) - [Basic Spark Package](https://george-jen.gitbook.io/data-science-and-apache-spark/basic-spark-package.md) - [Resilient Distributed Datasets (RDDs)](https://george-jen.gitbook.io/data-science-and-apache-spark/resilient-distributed-datasets-rdds.md) - [RDD Operations](https://george-jen.gitbook.io/data-science-and-apache-spark/rdd-operations.md) - [Passing Function to Spark](https://george-jen.gitbook.io/data-science-and-apache-spark/passing-function-to-spark.md) - [Printing elements of an RDD](https://george-jen.gitbook.io/data-science-and-apache-spark/printing-elements-of-an-rdd.md) - [Working with key value pair](https://george-jen.gitbook.io/data-science-and-apache-spark/working-with-key-value-pair.md) - [RDD Transformation Functions](https://george-jen.gitbook.io/data-science-and-apache-spark/rdd-transformation-funcitons.md) - [RDD Action Functions](https://george-jen.gitbook.io/data-science-and-apache-spark/rdd-action-functions.md) - [SPARK SQL](https://george-jen.gitbook.io/data-science-and-apache-spark/untitled-57.md) - [SQL](https://george-jen.gitbook.io/data-science-and-apache-spark/sql.md) - [Datasets and DataFrames](https://george-jen.gitbook.io/data-science-and-apache-spark/datasets-and-dataframes.md) - [SparkSession](https://george-jen.gitbook.io/data-science-and-apache-spark/sparksession.md) - [Creating DataFrames](https://george-jen.gitbook.io/data-science-and-apache-spark/creating-dataframes.md) - [Running SQL Queries Programmatically](https://george-jen.gitbook.io/data-science-and-apache-spark/running-sql-queries-programmatically.md) - [Issue from running Cartesian Join Query](https://george-jen.gitbook.io/data-science-and-apache-spark/issue-from-running-cartesian-join-query.md) - [Creating Datasets](https://george-jen.gitbook.io/data-science-and-apache-spark/creating-datasets.md) - [Interoperating with RDD](https://george-jen.gitbook.io/data-science-and-apache-spark/interoperating-with-rdd.md) - [Untyped User-Defined Aggregate Functions](https://george-jen.gitbook.io/data-science-and-apache-spark/untyped-user-defined-aggregate-functions.md) - [Generic Load/Save Functions](https://george-jen.gitbook.io/data-science-and-apache-spark/generic-load-save-functions.md) - [Manually specify file option](https://george-jen.gitbook.io/data-science-and-apache-spark/manually-specify-file-option.md) - [Run SQL on files directly](https://george-jen.gitbook.io/data-science-and-apache-spark/run-sql-on-files-directly.md) - [Save Mode](https://george-jen.gitbook.io/data-science-and-apache-spark/save-mode.md) - [Saving to Persistent Tables](https://george-jen.gitbook.io/data-science-and-apache-spark/saving-to-persistent-tables.md) - [Bucketing, Sorting and Partitioning](https://george-jen.gitbook.io/data-science-and-apache-spark/bucketing-sorting-and-partitioning.md) - [Apache Arrow](https://george-jen.gitbook.io/data-science-and-apache-spark/apache-arrow.md) - [Install Python Arrow Module PyArrow](https://george-jen.gitbook.io/data-science-and-apache-spark/install-python-arrow-module-pyarrow.md) - [Issue might happen import PyArrow](https://george-jen.gitbook.io/data-science-and-apache-spark/issue-might-happen-import-pyarrow.md) - [Enabling for Conversion to/from Pandas in Python](https://george-jen.gitbook.io/data-science-and-apache-spark/enabling-for-conversion-to-from-pandas.md) - [Connect to any data source the same consistent way](https://george-jen.gitbook.io/data-science-and-apache-spark/connect-to-any-data-source-the-same-consistent-way.md) - [Spark SQL Implementation Example in Scala](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-sql-implementation-example-in-scala.md) - [Run scala code in Eclipse IDE](https://george-jen.gitbook.io/data-science-and-apache-spark/run-scala-code-in-eclipse-ide.md) - [Hive Integration, run SQL or HiveQL queries on existing warehouses.](https://george-jen.gitbook.io/data-science-and-apache-spark/hive-integration-run-sql-or-hiveql-queries-on-existing-warehouses..md) - [Example: Enrich JSON](https://george-jen.gitbook.io/data-science-and-apache-spark/enrich-json.md) - [Integrate Tableau Data Visualization with Hive Data Warehouse and Apache Spark SQL](https://george-jen.gitbook.io/data-science-and-apache-spark/integrate-tableau-data-visualization-with-hive-data-warehouse-and-apache-spark-sql.md) - [Connect Tableau to Spark SQL running in VM with VirtualBox with NAT](https://george-jen.gitbook.io/data-science-and-apache-spark/connect-tableau-to-spark-sql-running-in-vm-with-virtualbox-with-nat.md) - [Issues with connecting from Tableau to Spark SQL](https://george-jen.gitbook.io/data-science-and-apache-spark/issues-with-connecting-from-tableau-to-spark-sql.md) - [SPARK Streaming](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-streaming.md) - [Discretized Streams (DStreams)](https://george-jen.gitbook.io/data-science-and-apache-spark/discretized-streams-dstreams.md) - [Transformations on DStreams](https://george-jen.gitbook.io/data-science-and-apache-spark/transformations-on-dstreams.md) - [map(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/map-func.md) - [filter(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/filter-func.md) - [repartition(numPartitions)](https://george-jen.gitbook.io/data-science-and-apache-spark/repartition-numpartitions.md) - [union(otherStream)](https://george-jen.gitbook.io/data-science-and-apache-spark/union-otherstream.md) - [reduce(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/reduce-func.md) - [count()](https://george-jen.gitbook.io/data-science-and-apache-spark/stream-count.md) - [countByValue()](https://george-jen.gitbook.io/data-science-and-apache-spark/countbyvalue.md) - [reduceByKey(func, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/reducebykey-func-numtasks.md) - [join(otherStream, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/join-otherstream-numtasks.md) - [cogroup(otherStream, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/cogroup-otherstream-numtasks.md) - [transform(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/transform-func.md) - [updateStateByKey(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/updatestatebykey-func.md) - [Scala Tips for updateStateByKey](https://george-jen.gitbook.io/data-science-and-apache-spark/scala-tips-for-updatestatebykey.md) - [repartition(numPartitions)](https://george-jen.gitbook.io/data-science-and-apache-spark/repartition-numpartitions-1.md) - [DStream Window Operations](https://george-jen.gitbook.io/data-science-and-apache-spark/dstream-window-operations.md) - [DStream Window Transformation](https://george-jen.gitbook.io/data-science-and-apache-spark/dstream-window-transformation.md) - [countByWindow(windowLength, slideInterval)](https://george-jen.gitbook.io/data-science-and-apache-spark/countbywindow-windowlength-slideinterval.md) - [reduceByWindow(func, windowLength, slideInterval)](https://george-jen.gitbook.io/data-science-and-apache-spark/reducebywindow-func-windowlength-slideinterval.md) - [reduceByKeyAndWindow(func, windowLength, slideInterval, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/reducebykeyandwindow-func-windowlength-slideinterval-numtasks.md) - [reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/reducebykeyandwindow-func-invfunc-windowlength-slideinterval-numtasks.md) - [countByValueAndWindow(windowLength, slideInterval, \[numTasks\])](https://george-jen.gitbook.io/data-science-and-apache-spark/countbyvalueandwindow-windowlength-slideinterval-numtasks.md) - [window(windowLength, slideInterval)](https://george-jen.gitbook.io/data-science-and-apache-spark/window-windowlength-slideinterval.md) - [Window DStream print(n)](https://george-jen.gitbook.io/data-science-and-apache-spark/window-dstream-print-n.md) - [saveAsTextFiles(prefix, \[suffix\])](https://george-jen.gitbook.io/data-science-and-apache-spark/untitled-92.md) - [saveAsObjectFiles(prefix, \[suffix\])](https://george-jen.gitbook.io/data-science-and-apache-spark/saveasobjectfiles-prefix-suffix.md) - [saveAsHadoopFiles(prefix, \[suffix\])](https://george-jen.gitbook.io/data-science-and-apache-spark/saveashadoopfiles-prefix-suffix.md) - [foreachRDD(func)](https://george-jen.gitbook.io/data-science-and-apache-spark/foreachrdd-func.md) - [Build Twitter Scala API Library for Spark Streaming using sbt](https://george-jen.gitbook.io/data-science-and-apache-spark/build-twitter-api-library-for-spark-streaming-using-sbt.md) - [Spark Streaming with Twitter, you can get public tweets by using Twitter API.](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-streaming-with-twitter-you-can-get-public-tweets-by-using-twitter-api..md) - [Spark streaming use case with Python](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-streaming-use-case-with-python.md) - [Spark Graph Computing](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-graph-computing.md) - [Spark Graph Computing Continue](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-graph-computing-continue.md) - [Graphx](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-1.md) - [Package org.apache.spark.graphx](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx.md) - [Edge Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edge-class.md) - [EdgeContext Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edgecontext-class.md) - [EdgeDirection Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edgedirection-class.md) - [EdgeRDD Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edgerdd-class.md) - [EdgeTriplet Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edgetriplet-class.md) - [Graph Class](https://george-jen.gitbook.io/data-science-and-apache-spark/graph-class.md) - [GraphLoader Object](https://george-jen.gitbook.io/data-science-and-apache-spark/graphloader-object.md) - [GraphOps Class](https://george-jen.gitbook.io/data-science-and-apache-spark/graphops-class.md) - [GraphXUtils Object](https://george-jen.gitbook.io/data-science-and-apache-spark/graphxutils-object.md) - [PartitionStrategy Trait](https://george-jen.gitbook.io/data-science-and-apache-spark/partitionstrategy-trait.md) - [Pregel Object](https://george-jen.gitbook.io/data-science-and-apache-spark/pregel-object.md) - [TripletFields Class](https://george-jen.gitbook.io/data-science-and-apache-spark/tripletfields-class.md) - [VertexRDD Class](https://george-jen.gitbook.io/data-science-and-apache-spark/vertexrdd-class.md) - [Package org.apache.spark.graphx.impl](https://george-jen.gitbook.io/data-science-and-apache-spark/untitled.md) - [AggregatingEdgeContext Class](https://george-jen.gitbook.io/data-science-and-apache-spark/aggregatingedgecontext-class.md) - [EdgeRDDImpl Class](https://george-jen.gitbook.io/data-science-and-apache-spark/edgerddimpl-class.md) - [Class GraphImpl\](https://george-jen.gitbook.io/data-science-and-apache-spark/class-graphimpl-less-than-vd-ed-greater-than.md) - [Class VertexRDDImpl\](https://george-jen.gitbook.io/data-science-and-apache-spark/class-vertexrddimpl-less-than-vd-greater-than.md) - [Package org.apache.spark.graphx.lib](https://george-jen.gitbook.io/data-science-and-apache-spark/package-org.apache.spark.graphx.lib-1.md) - [Class ConnectedComponents](https://george-jen.gitbook.io/data-science-and-apache-spark/class-connectedcomponents.md) - [Class LabelPropagation](https://george-jen.gitbook.io/data-science-and-apache-spark/class-labelpropagation.md) - [Class PageRank](https://george-jen.gitbook.io/data-science-and-apache-spark/class-pagerank.md) - [Class ShortestPaths](https://george-jen.gitbook.io/data-science-and-apache-spark/class-shortestpaths.md) - [Class StronglyConnectedComponents](https://george-jen.gitbook.io/data-science-and-apache-spark/class-stronglyconnectedcomponents.md) - [Class SVDPlusPlus](https://george-jen.gitbook.io/data-science-and-apache-spark/class-svdplusplus.md) - [Class SVDPlusPlus.Conf](https://george-jen.gitbook.io/data-science-and-apache-spark/class-svdplusplus.conf.md) - [Class TriangleCount](https://george-jen.gitbook.io/data-science-and-apache-spark/class-trianglecount.md) - [Package org.apache.spark.graphx.util](https://george-jen.gitbook.io/data-science-and-apache-spark/package-org.apache.spark.graphx.lib.md) - [Class BytecodeUtils](https://george-jen.gitbook.io/data-science-and-apache-spark/class-bytecodeutils.md) - [Class GraphGenerators](https://george-jen.gitbook.io/data-science-and-apache-spark/class-graphgenerators.md) - [Graphx Example 1](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-examples.md) - [Graphx Example 2](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-example-2.md) - [Graphx Example 3](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-example-3.md) - [Spark Graphx Describes Organization Chart Easy and Fast](https://george-jen.gitbook.io/data-science-and-apache-spark/untitled-98.md) - [Page Rank with Apache Spark Graphx](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-application-case-2.md) - [bulk synchronous parallel with Google Pregel Graphx Implementation Use Cases](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-application-case-3.md) - [Tree and Graph Traversal with and without Spark Graphx](https://george-jen.gitbook.io/data-science-and-apache-spark/tree-and-graph-traversal.md) - [Graphx Graph Traversal with Pregel Explained](https://george-jen.gitbook.io/data-science-and-apache-spark/graphx-graph-traversal-with-pregel-explained.md) - [Spark Machine Learning](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-machine-learning.md) - [Binary Classification](https://george-jen.gitbook.io/data-science-and-apache-spark/binary-classification.md) - [Multiclass Classification](https://george-jen.gitbook.io/data-science-and-apache-spark/multiclass-classification.md) - [Regression](https://george-jen.gitbook.io/data-science-and-apache-spark/regression.md) - [Correlation](https://george-jen.gitbook.io/data-science-and-apache-spark/correlation.md) - [Image Data Source](https://george-jen.gitbook.io/data-science-and-apache-spark/image-data-source.md) - [ML DataFrame is SQL DataFrame](https://george-jen.gitbook.io/data-science-and-apache-spark/ml-dataframe.md) - [ML Transformer](https://george-jen.gitbook.io/data-science-and-apache-spark/ml-transformer.md) - [ML Estimator](https://george-jen.gitbook.io/data-science-and-apache-spark/ml-estimator.md) - [ML Pipeline](https://george-jen.gitbook.io/data-science-and-apache-spark/ml-pipeline.md) - [Transformer/Estimator Parameters](https://george-jen.gitbook.io/data-science-and-apache-spark/transformer-estimator-parameters.md) - [Extracting, transforming and selecting features](https://george-jen.gitbook.io/data-science-and-apache-spark/extracting-transforming-and-selecting-features.md) - [TF-IDF](https://george-jen.gitbook.io/data-science-and-apache-spark/tf-idf.md) - [Word2Vec](https://george-jen.gitbook.io/data-science-and-apache-spark/word2vec.md) - [FeatureHasher](https://george-jen.gitbook.io/data-science-and-apache-spark/featurehasher.md) - [Tokenizer](https://george-jen.gitbook.io/data-science-and-apache-spark/tokenizer.md) - [CountVectorizer](https://george-jen.gitbook.io/data-science-and-apache-spark/countvectorizer.md) - [StopWordRemover](https://george-jen.gitbook.io/data-science-and-apache-spark/stopwordremover.md) - [n-gram](https://george-jen.gitbook.io/data-science-and-apache-spark/n-gram.md) - [Binarizer](https://george-jen.gitbook.io/data-science-and-apache-spark/binarizer.md) - [PCA](https://george-jen.gitbook.io/data-science-and-apache-spark/pca.md) - [PolynomialExpansion](https://george-jen.gitbook.io/data-science-and-apache-spark/polynomialexpansion.md) - [StringIndexer](https://george-jen.gitbook.io/data-science-and-apache-spark/stringindexer.md) - [Discrete Cosine Transform (DCT)](https://george-jen.gitbook.io/data-science-and-apache-spark/discrete-cosine-transform-dct.md) - [One-hot encoding](https://george-jen.gitbook.io/data-science-and-apache-spark/one-hot-encoding.md) - [StandardScaler](https://george-jen.gitbook.io/data-science-and-apache-spark/standardscaler.md) - [IndexToString](https://george-jen.gitbook.io/data-science-and-apache-spark/indextostring.md) - [VectorIndexer](https://george-jen.gitbook.io/data-science-and-apache-spark/vectorindexer.md) - [Interaction](https://george-jen.gitbook.io/data-science-and-apache-spark/interaction.md) - [Normalizer](https://george-jen.gitbook.io/data-science-and-apache-spark/normalizer.md) - [MinMaxScaler](https://george-jen.gitbook.io/data-science-and-apache-spark/minmaxscaler.md) - [MaxAbScaler](https://george-jen.gitbook.io/data-science-and-apache-spark/maxabscaler.md) - [Bucketizer](https://george-jen.gitbook.io/data-science-and-apache-spark/bucketizer.md) - [ElementwiseProduct](https://george-jen.gitbook.io/data-science-and-apache-spark/elementwiseproduct.md) - [SQLTransformer](https://george-jen.gitbook.io/data-science-and-apache-spark/sqltransformer.md) - [VectorAssembler](https://george-jen.gitbook.io/data-science-and-apache-spark/vectorassembler.md) - [VectorSizeHint](https://george-jen.gitbook.io/data-science-and-apache-spark/vectorsizehint.md) - [QuantileDiscretizer](https://george-jen.gitbook.io/data-science-and-apache-spark/quantilediscretizer.md) - [Imputer](https://george-jen.gitbook.io/data-science-and-apache-spark/imputer.md) - [VectorSlicer](https://george-jen.gitbook.io/data-science-and-apache-spark/vectorslicer.md) - [RFormula](https://george-jen.gitbook.io/data-science-and-apache-spark/rformula.md) - [ChiSqSelector](https://george-jen.gitbook.io/data-science-and-apache-spark/chisqselector.md) - [Locality Sensitive Hashing](https://george-jen.gitbook.io/data-science-and-apache-spark/locality-sensitive-hashing.md) - [MinHash for Jaccard Distance](https://george-jen.gitbook.io/data-science-and-apache-spark/minhash-for-jaccard-distance.md) - [Classification and Regression](https://george-jen.gitbook.io/data-science-and-apache-spark/classification-and-regression.md) - [LogisticRegression](https://george-jen.gitbook.io/data-science-and-apache-spark/logisticregression.md) - [OneVsRest](https://george-jen.gitbook.io/data-science-and-apache-spark/onevsrest.md) - [Naive Bayes classifiers](https://george-jen.gitbook.io/data-science-and-apache-spark/naive-bayes-classifiers.md) - [Decision trees](https://george-jen.gitbook.io/data-science-and-apache-spark/decision-trees.md) - [Random forests](https://george-jen.gitbook.io/data-science-and-apache-spark/random-forests.md) - [Gradient-boosted trees (GBTs)](https://george-jen.gitbook.io/data-science-and-apache-spark/gradient-boosted-trees-gbts.md) - [Multilayer perceptron classifier](https://george-jen.gitbook.io/data-science-and-apache-spark/multilayer-perceptron-classifier.md) - [Linear Support Vector Machine](https://george-jen.gitbook.io/data-science-and-apache-spark/linear-support-vector-machine.md) - [Linear Regression](https://george-jen.gitbook.io/data-science-and-apache-spark/linear-regression.md) - [Generalized linear regression](https://george-jen.gitbook.io/data-science-and-apache-spark/generalized-linear-regression.md) - [Isotonic regression](https://george-jen.gitbook.io/data-science-and-apache-spark/isotonic-regression.md) - [Decision Tree Regression](https://george-jen.gitbook.io/data-science-and-apache-spark/decision-tree-regression.md) - [Random Forest Regression](https://george-jen.gitbook.io/data-science-and-apache-spark/random-forest-regression.md) - [Gradient-boosted tree regression](https://george-jen.gitbook.io/data-science-and-apache-spark/gradient-boosted-tree-regression.md) - [Survival regression](https://george-jen.gitbook.io/data-science-and-apache-spark/survival-regression.md) - [Clustering](https://george-jen.gitbook.io/data-science-and-apache-spark/clustering.md) - [k-means](https://george-jen.gitbook.io/data-science-and-apache-spark/k-means.md) - [Latent Dirichlet allocation or LDA](https://george-jen.gitbook.io/data-science-and-apache-spark/latent-dirichlet-allocation-or-lda.md) - [Bisecting k-means](https://george-jen.gitbook.io/data-science-and-apache-spark/bisecting-k-means.md) - [A Gaussian Mixture Model](https://george-jen.gitbook.io/data-science-and-apache-spark/a-gaussian-mixture-model.md) - [Collaborative filtering](https://george-jen.gitbook.io/data-science-and-apache-spark/collaborative-filtering.md) - [Frequent Pattern Mining](https://george-jen.gitbook.io/data-science-and-apache-spark/frequent-pattern-mining.md) - [FP-Growth](https://george-jen.gitbook.io/data-science-and-apache-spark/fp-growth.md) - [PrefixSpan](https://george-jen.gitbook.io/data-science-and-apache-spark/prefixspan.md) - [ML Tuning: model selection and hyperparameter tuning](https://george-jen.gitbook.io/data-science-and-apache-spark/ml-tuning-model-selection-and-hyperparameter-tuning.md) - [Model selection (a.k.a. hyperparameter tuning)](https://george-jen.gitbook.io/data-science-and-apache-spark/model-selection-a.k.a.-hyperparameter-tuning.md) - [Cross-Validation](https://george-jen.gitbook.io/data-science-and-apache-spark/cross-validation.md) - [Train-Validation Split](https://george-jen.gitbook.io/data-science-and-apache-spark/train-validation-split.md) - [Spark Machine Learning Applications](https://george-jen.gitbook.io/data-science-and-apache-spark/spark-machine-learning-applications.md) - [Apache Spark SQL & Machine Learning on Genetic Variant Classifications](https://george-jen.gitbook.io/data-science-and-apache-spark/apache-spark-sql-and-machine-learning-on-genetic-variant-classifications.md) - [Data Visualization with Vegas Viz and Scala with Spark ML](https://george-jen.gitbook.io/data-science-and-apache-spark/data-visualization-with-vegas-viz-and-scala-with-spark-ml.md) - [Apache Spark Machine Learning with Dremio Data Lake Engine](https://george-jen.gitbook.io/data-science-and-apache-spark/apache-spark-machine-learning-with-dremio-data-lake-engine.md) - [Dremio Data Lake Engine Apache Arrow Flight Connector with Spark Machine Learning](https://george-jen.gitbook.io/data-science-and-apache-spark/dremio-data-lake-engine-apache-arrow-flight-connector-with-spark-machine-learning.md) - [Neural Network with Apache Spark Machine Learning Multilayer Perceptron Classifier](https://george-jen.gitbook.io/data-science-and-apache-spark/neural-network-with-apache-spark-machine-learning-multilayer-perceptron-classifier.md) - [Setup TensorFlow, Keras, Theano, Pytorch/torchvision on the CentOS VM](https://george-jen.gitbook.io/data-science-and-apache-spark/setup-tensorflow-keras-theano-pytorch-torchvision-on-the-centos-vm.md) - [Virus Xray Image Classification with Tensorflow Keras Python and Apache Spark Scala](https://george-jen.gitbook.io/data-science-and-apache-spark/virus-xray-image-classification-with-tensorflow-keras-python-and-apache-spark-scala.md) - [Appendix -- Video Presentations](https://george-jen.gitbook.io/data-science-and-apache-spark/appendix-video-presentations.md) - [References](https://george-jen.gitbook.io/data-science-and-apache-spark/references.md)