Data Science with Apache Spark

Search

⌃K

Data Science with Apache Spark

Search

⌃K

Basic Prerequisite Skills

Computer needed for this course

Spark Environment Setup

Dev environment setup, task list

Download and install Anaconda Python and create virtual environment with Python 3.6

Download and install Spark

Eclipse, the Scala IDE

Install findspark, add spylon-kernel for scala

ssh and scp client

Development environment on MacOS

Production Spark Environment Setup

VirtualBox only shows 32bit on AMD CPU

Configure VirtualBox NAT as Network Adapter on Guest VM and Allow putty ssh Through Port Forwarding

Docker deployment of Spark Cluster

Create customized Apache Spark Docker container

docker-compose and docker-compose.yml

Launch custom built Docker container with docker-compose

Entering Docker Container

Setup Hadoop, Hive and Spark on Linux without docker

Hadoop Preparation

Configure $HADOOP_HOME/etc/hadoop

Start and stop Hadoop

Work with Hadoop and HDFS file system

Connect to Hadoop web interface port 50070 and 8088

Initialize hive schema

Start hive metastore service.

Setup Apache Spark

Jupyter-notebook server

Python 3 Warm Up

Iterables/Collections

Conditional statement

Functions and methods

map and filter takes function as input

Input and if statement

Input from a file

Output to a file

Python coding exercise

Start Spylon-kernel on Jupyter-notebook

Type of Variable: Mutable or immutable

Block statement

Scala Data Type

Anonymous function

Scala map and filter methods

Scala if statement

Scala While Loop

Scala Exceptions + try catch finally

Scala coding exercise

Run a program to estimate pi

Common Spark command line

Run Scala code with spark-submit

Python with Apache Spark using Jupyter notebook

Spark Core Introduction

Spark and Scala Version

Basic Spark Package

Resilient Distributed Datasets (RDDs)

Passing Function to Spark

Printing elements of an RDD

Working with key value pair

RDD Transformation Functions

RDD Action Functions

Datasets and DataFrames

Creating DataFrames

Running SQL Queries Programmatically

Issue from running Cartesian Join Query

Creating Datasets

Interoperating with RDD

Untyped User-Defined Aggregate Functions

Generic Load/Save Functions

Manually specify file option

Run SQL on files directly

Saving to Persistent Tables

Bucketing, Sorting and Partitioning

Install Python Arrow Module PyArrow

Issue might happen import PyArrow

Enabling for Conversion to/from Pandas in Python

Connect to any data source the same consistent way

Spark SQL Implementation Example in Scala

Run scala code in Eclipse IDE

Hive Integration, run SQL or HiveQL queries on existing warehouses.

Example: Enrich JSON

Integrate Tableau Data Visualization with Hive Data Warehouse and Apache Spark SQL

Connect Tableau to Spark SQL running in VM with VirtualBox with NAT

Issues with connecting from Tableau to Spark SQL

SPARK Streaming

Discretized Streams (DStreams)

Transformations on DStreams

repartition(numPartitions)

union(otherStream)

reduceByKey(func, [numTasks])

join(otherStream, [numTasks])

cogroup(otherStream, [numTasks])

transform(func)

updateStateByKey(func)

Scala Tips for updateStateByKey

repartition(numPartitions)

DStream Window Operations

DStream Window Transformation

countByWindow(windowLength, slideInterval)

reduceByWindow(func, windowLength, slideInterval)

reduceByKeyAndWindow(func, windowLength, slideInterval, [numTasks])

reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval, [numTasks])

countByValueAndWindow(windowLength, slideInterval, [numTasks])

window(windowLength, slideInterval)

Window DStream print(n)

saveAsTextFiles(prefix, [suffix])

saveAsObjectFiles(prefix, [suffix])

saveAsHadoopFiles(prefix, [suffix])

foreachRDD(func)

Build Twitter Scala API Library for Spark Streaming using sbt

Spark Streaming with Twitter, you can get public tweets by using Twitter API.

Spark streaming use case with Python

Spark Graph Computing

Spark Graph Computing Continue

Package org.apache.spark.graphx

EdgeContext Class

EdgeDirection Class

EdgeTriplet Class

GraphLoader Object

GraphXUtils Object

PartitionStrategy Trait

TripletFields Class

VertexRDD Class

Package org.apache.spark.graphx.impl

AggregatingEdgeContext Class

EdgeRDDImpl Class

Class GraphImpl<VD,ED>

Class VertexRDDImpl<VD>

Package org.apache.spark.graphx.lib

Class ConnectedComponents

Class LabelPropagation

Class ShortestPaths

Class StronglyConnectedComponents

Class SVDPlusPlus

Class SVDPlusPlus.Conf

Class TriangleCount

Package org.apache.spark.graphx.util

Class BytecodeUtils

Class GraphGenerators

Graphx Example 1

Graphx Example 2

Graphx Example 3

Spark Graphx Describes Organization Chart Easy and Fast

Page Rank with Apache Spark Graphx

bulk synchronous parallel with Google Pregel Graphx Implementation Use Cases

Tree and Graph Traversal with and without Spark Graphx

Graphx Graph Traversal with Pregel Explained

Spark Machine Learning

Binary Classification

Multiclass Classification

Image Data Source

ML DataFrame is SQL DataFrame

Transformer/Estimator Parameters

Extracting, transforming and selecting features

CountVectorizer

StopWordRemover

PolynomialExpansion

Discrete Cosine Transform (DCT)

One-hot encoding

ElementwiseProduct

VectorAssembler

QuantileDiscretizer

Locality Sensitive Hashing

MinHash for Jaccard Distance

Classification and Regression

LogisticRegression

Naive Bayes classifiers

Gradient-boosted trees (GBTs)

Multilayer perceptron classifier

Linear Support Vector Machine

Linear Regression

Generalized linear regression

Isotonic regression

Decision Tree Regression

Random Forest Regression

Gradient-boosted tree regression

Survival regression

Latent Dirichlet allocation or LDA

Bisecting k-means

A Gaussian Mixture Model

Collaborative filtering

Frequent Pattern Mining

ML Tuning: model selection and hyperparameter tuning

Model selection (a.k.a. hyperparameter tuning)

Cross-Validation

Train-Validation Split

Spark Machine Learning Applications

Apache Spark SQL & Machine Learning on Genetic Variant Classifications

Data Visualization with Vegas Viz and Scala with Spark ML

Apache Spark Machine Learning with Dremio Data Lake Engine

Dremio Data Lake Engine Apache Arrow Flight Connector with Spark Machine Learning

Neural Network with Apache Spark Machine Learning Multilayer Perceptron Classifier

Setup TensorFlow, Keras, Theano, Pytorch/torchvision on the CentOS VM

Virus Xray Image Classification with Tensorflow Keras Python and Apache Spark Scala

Appendix -- Video Presentations

Powered By GitBook

Multiclass Classification

Multiclass Classification
decision trees
naive Bayes

Binary Classification

Last modified 3yr ago