⌘Ctrlk

Preface
Contents
Basic Prerequisite Skills
Computer needed for this course
Spark Environment Setup
Dev environment setup, task list
JDK setup
Download and install Anaconda Python and create virtual environment with Python 3.6
Download and install Spark
Eclipse, the Scala IDE
Install findspark, add spylon-kernel for scala
ssh and scp client
Summary
Development environment on MacOS
Production Spark Environment Setup
VirtualBox VM
VirtualBox only shows 32bit on AMD CPU
Configure VirtualBox NAT as Network Adapter on Guest VM and Allow putty ssh Through Port Forwarding
Docker deployment of Spark Cluster
Create customized Apache Spark Docker container
Dockerfile
docker-compose and docker-compose.yml
Launch custom built Docker container with docker-compose
Entering Docker Container
Setup Hadoop, Hive and Spark on Linux without docker
Hadoop Preparation
Hadoop setup
Configure $HADOOP_HOME/etc/hadoop
HDFS
Start and stop Hadoop
Work with Hadoop and HDFS file system
Connect to Hadoop web interface port 50070 and 8088
Install Hive
hive home
Initialize hive schema
Start hive metastore service.
hive-site.xml
Hive client
Setup Apache Spark
Spark Home
Jupyter-notebook server
Python 3 Warm Up
Basics
Iterables/Collections
Strings
List
Tuple
Dictionary
Set
Conditional statement
for loop
while loop
Functions and methods
map and filter
map and filter takes function as input
lambda
Python Class
Input and if statement
Input from a file
Output to a file
try except
Python coding exercise
Scala Warm Up
Start Spylon-kernel on Jupyter-notebook
Type of Variable: Mutable or immutable
Block statement
Scala Data Type
Array in Scala
Methods
Functions
Anonymous function
Scala map and filter methods
Class
Objects
Trait
Tuple in Scala
List/Seq
Set in Scala
Scala Map
Scala if statement
Scala for loop
Scala While Loop
Scala Exceptions + try catch finally
Scala coding exercise
Run a program to estimate pi
Common Spark command line
Run Scala code with spark-submit
Python with Apache Spark using Jupyter notebook
Spark Core Introduction
Spark and Scala Version
Basic Spark Package
Resilient Distributed Datasets (RDDs)
RDD Operations
Passing Function to Spark
Printing elements of an RDD
Working with key value pair
RDD Transformation Functions
RDD Action Functions
SPARK SQL
SQL
Datasets and DataFrames
SparkSession
Creating DataFrames
Running SQL Queries Programmatically
Issue from running Cartesian Join Query
Creating Datasets
Interoperating with RDD
Untyped User-Defined Aggregate Functions
Generic Load/Save Functions
Manually specify file option
Run SQL on files directly
Save Mode
Saving to Persistent Tables
Bucketing, Sorting and Partitioning
Apache Arrow
Install Python Arrow Module PyArrow
Issue might happen import PyArrow
Enabling for Conversion to/from Pandas in Python
Connect to any data source the same consistent way
Spark SQL Implementation Example in Scala
Run scala code in Eclipse IDE
Hive Integration, run SQL or HiveQL queries on existing warehouses.
Example: Enrich JSON
Integrate Tableau Data Visualization with Hive Data Warehouse and Apache Spark SQL
Connect Tableau to Spark SQL running in VM with VirtualBox with NAT
Issues with connecting from Tableau to Spark SQL
SPARK Streaming
Discretized Streams (DStreams)
Transformations on DStreams
map(func)
filter(func)
repartition(numPartitions)
union(otherStream)
reduce(func)
count()
countByValue()
reduceByKey(func, [numTasks])
join(otherStream, [numTasks])
cogroup(otherStream, [numTasks])
transform(func)
updateStateByKey(func)
Scala Tips for updateStateByKey
repartition(numPartitions)
DStream Window Operations
DStream Window Transformation
countByWindow(windowLength, slideInterval)
reduceByWindow(func, windowLength, slideInterval)
reduceByKeyAndWindow(func, windowLength, slideInterval, [numTasks])
reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, [numTasks])
countByValueAndWindow(windowLength, slideInterval, [numTasks])
window(windowLength, slideInterval)
Window DStream print(n)
saveAsTextFiles(prefix, [suffix])
saveAsObjectFiles(prefix, [suffix])
saveAsHadoopFiles(prefix, [suffix])
foreachRDD(func)
Build Twitter Scala API Library for Spark Streaming using sbt
Spark Streaming with Twitter, you can get public tweets by using Twitter API.
Spark streaming use case with Python
Spark Graph Computing
Spark Graph Computing Continue
Graphx
Package org.apache.spark.graphx
Edge Class
EdgeContext Class
EdgeDirection Class
EdgeRDD Class
EdgeTriplet Class
Graph Class
GraphLoader Object
GraphOps Class
GraphXUtils Object
PartitionStrategy Trait
Pregel Object
TripletFields Class
VertexRDD Class
Package org.apache.spark.graphx.impl
AggregatingEdgeContext Class
EdgeRDDImpl Class
Class GraphImpl<VD,ED>
Class VertexRDDImpl<VD>
Package org.apache.spark.graphx.lib
Class ConnectedComponents
Class LabelPropagation
Class PageRank
Class ShortestPaths
Class StronglyConnectedComponents
Class SVDPlusPlus
Class SVDPlusPlus.Conf
Class TriangleCount
Package org.apache.spark.graphx.util
Class BytecodeUtils
Class GraphGenerators
Graphx Example 1
Graphx Example 2
Graphx Example 3
Spark Graphx Describes Organization Chart Easy and Fast
Page Rank with Apache Spark Graphx
bulk synchronous parallel with Google Pregel Graphx Implementation Use Cases
Tree and Graph Traversal with and without Spark Graphx
Graphx Graph Traversal with Pregel Explained
Spark Machine Learning
Binary Classification
Multiclass Classification
Regression
Correlation
Image Data Source
ML DataFrame is SQL DataFrame
ML Transformer
ML Estimator
ML Pipeline
Transformer/Estimator Parameters
Extracting, transforming and selecting features
TF-IDF
Word2Vec
FeatureHasher
Tokenizer
CountVectorizer
StopWordRemover
n-gram
Binarizer
PCA
PolynomialExpansion
StringIndexer
Discrete Cosine Transform (DCT)
One-hot encoding
StandardScaler
IndexToString
VectorIndexer
Interaction
Normalizer
MinMaxScaler
MaxAbScaler
Bucketizer
ElementwiseProduct
SQLTransformer
VectorAssembler
VectorSizeHint
QuantileDiscretizer
Imputer
VectorSlicer
RFormula
ChiSqSelector
Locality Sensitive Hashing
MinHash for Jaccard Distance
Classification and Regression
LogisticRegression
OneVsRest
Naive Bayes classifiers
Decision trees
Random forests
Gradient-boosted trees (GBTs)
Multilayer perceptron classifier
Linear Support Vector Machine
Linear Regression
Generalized linear regression
Isotonic regression
Decision Tree Regression
Random Forest Regression
Gradient-boosted tree regression
Survival regression
Clustering
k-means
Latent Dirichlet allocation or LDA
Bisecting k-means
A Gaussian Mixture Model
Collaborative filtering
Frequent Pattern Mining
FP-Growth
PrefixSpan
ML Tuning: model selection and hyperparameter tuning
Model selection (a.k.a. hyperparameter tuning)
Cross-Validation
Train-Validation Split
Spark Machine Learning Applications
Apache Spark SQL & Machine Learning on Genetic Variant Classifications
Data Visualization with Vegas Viz and Scala with Spark ML
Apache Spark Machine Learning with Dremio Data Lake Engine
Dremio Data Lake Engine Apache Arrow Flight Connector with Spark Machine Learning
Neural Network with Apache Spark Machine Learning Multilayer Perceptron Classifier
Setup TensorFlow, Keras, Theano, Pytorch/torchvision on the CentOS VM
Virus Xray Image Classification with Tensorflow Keras Python and Apache Spark Scala
Appendix -- Video Presentations
References

For the complete documentation index, see llms.txt. This page is also available as Markdown.

Transformations on DStreams

Similar to that of RDDs, transformations allow the data from the input DStream to be modified. DStreams support many of the transformations available on normal Spark RDD’s. Some of the common ones are as following list of methods.

PreviousDiscretized Streams (DStreams)Nextmap(func)

Last updated 6 years ago