MCS

Big Data

Statistical Machine Learning

(CSE 575)
Note: The information below and course outline are subject to modifications and updates.

About this Course

The link between inference and computation is central to statistical machine learning, which
combines the computational sciences with statistics. In addition to artificial intelligence, fields
such as information management, finance, bioinformatics, and communications are significantly
influenced by developments in statistical machine learning. This course investigates the
data mining and statistical pattern recognition that support artificial intelligence. Main topics
covered include supervised learning; unsupervised learning; and deep learning, including major
components of machine learning and the data analytics that enable it.

Specific topics covered include:

yy Probability distributions yy Clustering
yy Maximum likelihood estimation yy Principal component analysis
yy Naive Bayes yy Neural networks
yy Logistic regression yy Convolutional neural networks
yy Support vector machines

Required Prior Knowledge and Skills

yyBasics of linear algebra, statistics, calculus, and algorithm design and analysis
yyProgramming (language such as Python or MATLAB)

Learning Outcomes

Learners completing this course will be able to:

yyDistinguish between supervised learning and unsupervised learning

yyApply common probability distributions in machine learning applications
yyUse cross validation to select parameters
yyUse maximum likelihood estimate (MLE) for parameter estimation
yyImplement fundamental learning algorithms such as logistic regression and k-means clustering

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 1
MCS
Big Data

yy Implement more advanced learning algorithms such as support vector machines and
convolutional neural networks
yy Design a deep network using an exemplar application to solve a specific problem

yy Apply key techniques employed in building deep learning architectures

Projects
yy Project 1: TBD
yy Project 2: TBD
yy Project 2: Implement k-means algorithm and its variants on a real document clustering task
yy Project 3: Image Classification using a standard dataset to train a deep network to classify
images and evaluate the performance on both the training set and the test set

Course Content

Instruction
yy Video lectures
yy Other videos
yy Readings
yy Interactive learning objects
yy Live office hours

Assessments
yy Practice activities and quizzes (auto-graded)
yy Practice assignments (instructor- or peer-reviewed)
yy Team and/or individual project(s) (instructor-graded)
yy Midterm or final exam (proctored, graded)

Estimated Workload/ Time Commitment Per Week

Approximately 15-20 hours per week.

Technology Requirements

Hardware
yy Standard with major OS

Software and Other

yy Standard - technology integrations will be provided through Coursera

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 2
MCS
Big Data

Course Outline
Unit 1: Introduction to Machine Learning

Learning Objectives
1.1 Describe common misconceptions of machine learning
1.2 Define machine learning
1.3 Distinguish between supervised learning and unsupervised learning
1.4 Compare numerical and graphical data representations
1.5 Describe applications of machine learning

Module 1: Defining Machine Learning

Common misconceptions
What is Machine Learning?
Related fields

Module 2: Styles of Machine Learning

Supervised learning
Unsupervised learning

Module 3: Data Representations

Data representation
Numerical representation
Graph representation

Module 4: Applications of Machine Learning

Recognizing examples
Familiar applications
Emerging applications

Unit 2: Statistical Core of Machine Learning

Learning Objectives
2.1 Apply common probability distributions in machine learning applications
2.2 Use maximum likelihood estimate (MLE) for parameter estimation

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 3
MCS
Big Data

Module 1: Probability
Discrete Random Variables
Probability Mass Function (PMF)
Common Distributions of PMF
• Uniform
• Binomial
Joint Probability Mass Function
Conditional Probability
Relationship Between Marginal and Joint Probability
Bayes Theorem
Independent Random Variables
Continuous Random Variables
Probability Density Function (PDF)
Common Distributions of PDF
• Normal
• Beta
Joint Probability Density Function
Moments of Random Variables

Module 2: Maximum Likelihood Estimation

Likelihood function
• For discrete probability distribution
• For continuous probability distribution
Maximum likelihood estimation
• For discrete probability distribution
• For continuous probability distribution
• For mean and standard deviation

Unit 3: Supervised Learning: Two Models

Learning Objectives
3.1 Differentiate between generative and discriminative models for supervised learning
3.2 Implement fundamental learning algorithms such as Naive Bayes and Logistic Regression
3.3 Interpret empirical comparisons of Naive Bayes and Logistic Regression

Module 1: Generative vs Discriminative Model of Supervised Learning

Generative vs Discriminative models for supervised learning
• Essential distinction
• Generative model: Naive Bayes
• Discriminative model: Logistic Regression

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 4
MCS
Big Data

Module 2: Naive Bayes

• Naive Bayes Assumption
• Decision Rule
Parameters of Naive Bayes
Maximum Likelihood Estimation (MLE) for Naive Bayes Parameters
Text Classification using Naive Bayes
• Bag of Words Model for Text

Module 3: Logistic Regression

Logistic Function
Linear Classifier
Parameter Estimation
Maximizing Conditional Log Likelihood
Gradient Ascent
Optimization Algorithm

Module 4: Comparing the Models

Empirical Comparison of Naive Bayes and Logistic Regression

Unit 4: Supervised Learning: Support Vector Machines

Learning Objectives
4.1 Differentiate between linearly separable and non-separable support vector machines
4.2 Explain the role of the kernel trick in support vector machines
4.3 Explain options for picking magic parameters in support vector machines
4.4 Implement the more advanced learning algorithm known as support vector machines

Module 1: Introduction to Support Vector Machines

SVM: Separable vs non-separable

Module 2: Separable
Linearly Separable Example
Max-margin Separating Hyperplane
Margin Maximization with Canonical Hyperplanes
Optimization Problem of SVM: separable case
Dual SVM Formulation: separable case

Module 3: Non-separable
Linearly Non-separable Example
Hinge Loss
Optimization Problem of SVM: non-separable case
Dual SVM Formulation: non-separable case
Input Space to Feature Space

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 5
MCS
Big Data

Kernel Trick
• Common Kernels
• Test Example
• SVM with the Kernel Trick

Module 4: Parameter Selection

How to Pick the Magic Parameters?
• Option #1: Leave-One-Out Cross Validation (LOOCV)
• Option #2: Cross Validation

Unit 5: Unsupervised Learning: Clustering

Learning Objectives
5.1 Differentiate between clustering in supervised vs. unsupervised learning
5.2 Explain how to efficiently cluster data
5.3 Apply the k-means algorithm
5.4 Explain the relationship between the several K-means variants

Module 1: Introduction to Clustering

The role of clustering in machine learning
Clustering in supervised versus unsupervised learning
How to find good clustering
• Intuition
• An example
• Mathematical formulation
How to efficiently cluster data
• Challenge - combinatorial nature
• Solution:
• High-level Idea:alternation
• Details - step 1: fix the cluster clusters, find the cluster membership
• Details - step 2: fix the cluster membership, update the cluster center

Module 2: K-means
K-means for clustering
K-means models
Properties of the K-means algorithm
• Initialization
• fix the cluster clusters, find the cluster membership
• fix the cluster membership, update the cluster center
• Repeat the above two steps until convergence
Comparing K-means clusterings

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 6
MCS
Big Data

A Numerical Example
• Input data, plot them in 1-d space
• Pick the initial cluster centers
• Run k-means algorithm one iteration
• Show how the cluster membership changes
• Show how the cluster centrs change
K-means algorithm considerations

Module 3: K-means Variants

K-means as matrix factorization
The k-means problem
• Input of k-means
• Mathematical formulation
• Two special case (k=1 vs. k=n)
Hardness of K-means problem
• When d>2, k-means is NP-hard
• When d=1, k-means is polynomially solvable
Optimality of Kmeans
• In general, it only finds a local optimum
• Convergence of kmeans
• The impact of initial cluster centers
• A numerical example about the impact of initial cluster centers
• Impact of outlier
Alternatives to random initialization
• Multiple runs
• kmeans++

Unit 6: Unsupervised Learning: Dimensionality Reduction

Learning Objectives
6.1 Illustrate the process of dimensionality reduction
6.2 Apply the PCA algorithm
6.3 Explain the relationship between PCA and SVD

Module 1: Introduction to Dimensionality Reduction

What is dimensionality reduction?
The role of dimensionality reduction in machine learning

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 7
MCS
Big Data

Module 2: Using Principal Component Analysis (PCA)

Introduction to using PCA
• Inputs of PCA
• Outputs of PCA
• A Numerical example
Maximizing the projected variance for the numerical example (d=1)
• How to calculate the projected data using original data and projection direction
• How to calculate the projected mean
• How to calculate projected variance
Maximizing the projected variance for the general case (d=1)
• One projected data
• Projected sample mean
• Sample variance matrix
• projected variance
Optimization formulation for PCA (d=1)
• Objective function
• Constraint & why we need it
• Optimization variable
Solving the optimization problem for PCA (d=1)
• Overall strategy: lagrangian
• Step 1: write down the lagrangian function
• Step 2: calculate the partial derivative
• Step 3: set the partial derivative to zero
• Step 4: plug in step 3 back to the objective function J
• Step 5: seek for the largest eigenvalue of S
Solving the optimization problem for PCA (d>1)
• Fact: d principle components are the first d eigenvectors of the sample variance matrix S
• Prove it by induction
• Step 0: Base case
• Step 1: projected variance when d>1
• Step 2: the optimization formulation
• Step 3: solve the optimization problem using lagrangian
Minimizing the reconstruction error
• Input data
• Projected data
• Reconstruction error
• Minimizing reconstruction error = maximizing projected variance
A matrix representation for minimizing reconstruction error
• Assumption
• Input data matrix
• Projected data matrix
• PC matrix
• Objective function

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 8
MCS
Big Data

PCA versus SVD

• Assumption
• Input data matrix X
• SVD of X
• Left singular matrix = projected data matrix
• Singular value matrix and right singular vector matrix = PC matrix
PCA versus Feature Selection
• Input data matrix
• Rows of input data matrix
• Columns of input data matrix
• Two key points of PCA
• Un-supervised learning
• Generate a few new features
• Two key points of feature selection
• Typically supervised learning
• Select a few original features

Unit 7: Deep Learning: Key Techniques

Learning Objectives
7.1: Describe the big-picture view of how neural networks work.
7.2: Identify the basic building blocks and notations of deep neural networks.
7.3: Explain how in principle learning is achieved in a deep network.
7.4: Explain key techniques that enable efficient learning in deep networks.
7.5: Appraise the detailed architecture of a basic convolutional neural network.
7.6: Compare the basic concepts and corresponding architecture for recurrent neural networks
and autoencoders.

Module 1: Introduction to Dimensionality Reduction

Brief historical view of artificial neural network and deep learning
Early models of artificial neural network and their learning algorithms
Deep learning: what it is and what it is not

Module 2: Key Techniques Enabling Deep Learning

Back-propagation algorithm for learning
Choice of activation functions
A few regularization methods

Module 3: Some Basic Deep Architecture

Convolutional Neural Network
Recurrent Neural Networks
Autoencoders

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 9
MCS
Big Data

Unit 8: Deep Learning: Exemplar Applications

Learning Outcomes
8.1: Appraise image classification for deep learning
8.2: Appraise video-based inference for deep learning
8.3: Appraise Generative Adversarial Networks (GANs) for deep learning
8.4: Design a deep network using an exemplar application to solve a specific problem

Module 1: Image Classification

A typical network architecture used for image classification
Parameters for defining an image classification network
Common tricks for improving classification performance

Module 2: Video-Based Inference

Challenges in using deep networks for sequential data
Difference between image-based and video-based classification
Using video action recognition to contrast the difference between these classification tasks
A sample network for video-based inference

Module 3: Generative Adversarial Networks ( GANs)

Basic concepts behind GANs
GANS variants and their applications

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 10
MCS
Big Data

About ASU

Established in Tempe in 1885, Arizona State University (ASU) has developed a new model
for the American Research University, creating an institution that is committed to access,
excellence and impact.

As the prototype for a New American University, ASU pursues research that contributes to the
public good, and ASU assumes major responsibility for the economic, social and cultural vitality
of the communities that surround it. Recognizing the university’s groundbreaking initiatives,
partnerships, programs and research, U.S. News and World Report has named ASU as the most
innovative university all three years it has had the category.

The innovation ranking is due at least in part to a more than 80 percent improvement in ASU’s
graduation rate in the past 15 years, the fact that ASU is the fastest-growing research university
in the country and the emphasis on inclusion and student success that has led to more than 50
percent of the school’s in-state freshman coming from minority backgrounds.

About Ira A. Fulton Schools of Engineering

Structured around grand challenges and improving the quality of life on a global scale, the Ira
A. Fulton Schools of Engineering at Arizona State University integrates traditionally separate
disciplines and supports collaborative research in the multidisciplinary areas of biological and
health systems; sustainable engineering and the built environment; matter, transport and energy;
and computing and decision systems. As the largest engineering program in the United States,
students can pursue their educational and career goals through 25 undergraduate degrees or 39
graduate programs and rich experiential education offerings. The Fulton Schools are dedicated
to engineering programs that combine a strong core foundation with top faculty and a reputation
for graduating students who are aggressively recruited by top companies or become superior
candidates for graduate studies in medicine, law, engineering and science.

About the School of Computing, Informatics, & Decision Systems Engineering

The School of Computing, Informatics, and Decision Systems Engineering advances

developments and innovation in artificial intelligence, big data, cybersecurity and digital forensics,
and software engineering. Our faculty are winning prestigious honors in professional societies,
resulting in leadership of renowned research centers in homeland security operational efficiency,
data engineering, and cybersecurity and digital forensics. The school’s rapid growth of student
enrollment isn’t limited to the number of students at ASU’s Tempe and Polytechnic campuses as
it continues to lead in online education. In addition to the Online Master of Computer Science,
the school also offers an Online Bachelor of Science in Software Engineering, and the first four-
year, completely online Bachelor of Science in Engineering program in engineering management.

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 11
MCS
Big Data

Creators
Jingrui He is an assistant professor in the School of Computing, Informatics, and Decision
Systems Engineering at Arizona State University. She received her Ph.D. from Carnegie
Mellon University. She joined ASU in 2014 and directs the Statistical Learning Lab (STAR
Lab). Her research focuses on rare category analysis, heterogeneous machine learning,
active learning and semi-supervised learning, with applications in social media analysis,
healthcare, manufacturing process, etc.

Baoxin Li is currently a professor and the chair of the Computer Science & Engineering
Program and a Graduate Faculty Endorsed to Chair in the Electrical Engineering and
Computer Engineering programs. From 2000 to 2004, he was a Senior Researcher with
SHARP Laboratories of America, where he was the technical lead in developing SHARP’s
HiIMPACT Sports™ technologies. He was also an Adjunct Professor with the Portland State
University from 2003 to 2004. His general research interests are on visual computing and
machine learning, especially their application in the context of human-centered computing.

Hanghang Tong is currently an assistant professor at School of Computing, Informatics,

and Decision Systems Engineering (CIDSE), Arizona State University since August 2014.
Before that,he was an assistant professor at Computer Science Department, City College,
City University of New York, a research staff member at IBM T.J. Watson Research Center
and a Post-doctoral fellow in Carnegie Mellon University. His research interest is in large
scale data mining for graphs and multimedia.

Statistical Machine Learning

Lead: Jingrui He | Updated 6/28/2018 12
AOL-4715