Learning PySpark
By Tomasz Drabas and Denny Lee
()
About this ebook
- Learn why and how you can efficiently use Python to process data and build machine learning models in Apache Spark 2.0
- Develop and deploy efficient, scalable real-time Spark solutions
- Take your understanding of using Spark with Python to the next level with this jump start guide
If you are a Python developer who wants to learn about the Apache Spark 2.0 ecosystem, this book is for you. A firm understanding of Python is expected to get the best out of the book. Familiarity with Spark would be useful, but is not mandatory.
Read more from Tomasz Drabas
Practical Data Analysis Cookbook Rating: 0 out of 5 stars0 ratingsPySpark Cookbook: Over 60 recipes for implementing big data processing and analytics using Apache Spark and Python Rating: 0 out of 5 stars0 ratings
Related to Learning PySpark
Related ebooks
Data Analysis with Python and PySpark Rating: 0 out of 5 stars0 ratingsLearning Apache Spark 2 Rating: 0 out of 5 stars0 ratingsMLOps Engineering at Scale Rating: 0 out of 5 stars0 ratingsMastering Spark for Data Science Rating: 0 out of 5 stars0 ratingsSpark in Action: Covers Apache Spark 3 with Examples in Java, Python, and Scala Rating: 0 out of 5 stars0 ratingsFast Data Processing with Spark 2 - Third Edition Rating: 0 out of 5 stars0 ratingsBuilding Web Applications with Python and Neo4j Rating: 0 out of 5 stars0 ratingsFrank Kane's Taming Big Data with Apache Spark and Python Rating: 0 out of 5 stars0 ratingsData Lake Development with Big Data Rating: 0 out of 5 stars0 ratingsData Pipelines with Apache Airflow Rating: 0 out of 5 stars0 ratingsDatabricks A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsReal-Time Big Data Analytics Rating: 5 out of 5 stars5/5Mastering Machine Learning on AWS: Advanced machine learning in Python using SageMaker, Apache Spark, and TensorFlow Rating: 0 out of 5 stars0 ratingsData Lake for Enterprises Rating: 0 out of 5 stars0 ratingsPractical Full Stack Machine Learning: A Guide to Build Reliable, Reusable, and Production-Ready Full Stack ML Solutions Rating: 0 out of 5 stars0 ratingsApache Spark for Data Science Cookbook Rating: 0 out of 5 stars0 ratingsMachine Learning with Spark - Second Edition Rating: 0 out of 5 stars0 ratingsData Engineering on Azure Rating: 0 out of 5 stars0 ratingsHadoop Real-World Solutions Cookbook - Second Edition Rating: 0 out of 5 stars0 ratingsApache Spark 2.x Cookbook Rating: 0 out of 5 stars0 ratingsLearning pandas Rating: 4 out of 5 stars4/5Big Data Analytics Rating: 0 out of 5 stars0 ratingsAzure Storage, Streaming, and Batch Analytics: A guide for data engineers Rating: 0 out of 5 stars0 ratingsPython Data Analysis - Second Edition Rating: 0 out of 5 stars0 ratingsHands-On Data Analysis with Pandas: Efficiently perform data collection, wrangling, analysis, and visualization using Python Rating: 0 out of 5 stars0 ratingsThe Data Science Workshop: A New, Interactive Approach to Learning Data Science Rating: 0 out of 5 stars0 ratingsOperationalizing Machine Learning Pipelines: Building Reusable and Reproducible Machine Learning Pipelines Using MLOps Rating: 0 out of 5 stars0 ratings
Computers For You
The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution Rating: 4 out of 5 stars4/5Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics Rating: 4 out of 5 stars4/5Elon Musk Rating: 4 out of 5 stars4/5Mastering ChatGPT: 21 Prompts Templates for Effortless Writing Rating: 5 out of 5 stars5/5The Invisible Rainbow: A History of Electricity and Life Rating: 5 out of 5 stars5/5Slenderman: Online Obsession, Mental Illness, and the Violent Crime of Two Midwestern Girls Rating: 4 out of 5 stars4/5The Hacker Crackdown: Law and Disorder on the Electronic Frontier Rating: 4 out of 5 stars4/5101 Awesome Builds: Minecraft® Secrets from the World's Greatest Crafters Rating: 4 out of 5 stars4/5CompTIA Security+ Get Certified Get Ahead: SY0-701 Study Guide Rating: 5 out of 5 stars5/5Uncanny Valley: A Memoir Rating: 4 out of 5 stars4/5SQL QuickStart Guide: The Simplified Beginner's Guide to Managing, Analyzing, and Manipulating Data With SQL Rating: 4 out of 5 stars4/5Ultimate Guide to Mastering Command Blocks!: Minecraft Keys to Unlocking Secret Commands Rating: 5 out of 5 stars5/5Deep Search: How to Explore the Internet More Effectively Rating: 5 out of 5 stars5/5Procreate for Beginners: Introduction to Procreate for Drawing and Illustrating on the iPad Rating: 0 out of 5 stars0 ratingsHow to Create Cpn Numbers the Right way: A Step by Step Guide to Creating cpn Numbers Legally Rating: 4 out of 5 stars4/5Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Rating: 4 out of 5 stars4/5Grokking Algorithms: An illustrated guide for programmers and other curious people Rating: 4 out of 5 stars4/5The ChatGPT Millionaire Handbook: Make Money Online With the Power of AI Technology Rating: 4 out of 5 stars4/5Creating Online Courses with ChatGPT | A Step-by-Step Guide with Prompt Templates Rating: 4 out of 5 stars4/5Dark Aeon: Transhumanism and the War Against Humanity Rating: 5 out of 5 stars5/5CompTIA IT Fundamentals (ITF+) Study Guide: Exam FC0-U61 Rating: 0 out of 5 stars0 ratingsThe Professional Voiceover Handbook: Voiceover training, #1 Rating: 5 out of 5 stars5/5
Reviews for Learning PySpark
0 ratings0 reviews
Book preview
Learning PySpark - Tomasz Drabas
Table of Contents
Learning PySpark
Credits
Foreword
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
1. Understanding Spark
What is Apache Spark?
Spark Jobs and APIs
Execution process
Resilient Distributed Dataset
DataFrames
Datasets
Catalyst Optimizer
Project Tungsten
Spark 2.0 architecture
Unifying Datasets and DataFrames
Introducing SparkSession
Tungsten phase 2
Structured Streaming
Continuous applications
Summary
2. Resilient Distributed Datasets
Internal workings of an RDD
Creating RDDs
Schema
Reading from files
Lambda expressions
Global versus local scope
Transformations
The .map(...) transformation
The .filter(...) transformation
The .flatMap(...) transformation
The .distinct(...) transformation
The .sample(...) transformation
The .leftOuterJoin(...) transformation
The .repartition(...) transformation
Actions
The .take(...) method
The .collect(...) method
The .reduce(...) method
The .count(...) method
The .saveAsTextFile(...) method
The .foreach(...) method
Summary
3. DataFrames
Python to RDD communications
Catalyst Optimizer refresh
Speeding up PySpark with DataFrames
Creating DataFrames
Generating our own JSON data
Creating a DataFrame
Creating a temporary table
Simple DataFrame queries
DataFrame API query
SQL query
Interoperating with RDDs
Inferring the schema using reflection
Programmatically specifying the schema
Querying with the DataFrame API
Number of rows
Running filter statements
Querying with SQL
Number of rows
Running filter statements using the where Clauses
DataFrame scenario – on-time flight performance
Preparing the source datasets
Joining flight performance and airports
Visualizing our flight-performance data
Spark Dataset API
Summary
4. Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Duplicates
Missing observations
Outliers
Getting familiar with your data
Descriptive statistics
Correlations
Visualization
Histograms
Interactions between features
Summary
5. Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Descriptive statistics
Correlations
Statistical testing
Creating the final dataset
Creating an RDD of LabeledPoints
Splitting into training and testing
Predicting infant survival
Logistic regression in MLlib
Selecting only the most predictable features
Random forest in MLlib
Summary
6. Introducing the ML Package
Overview of the package
Transformer
Estimators
Classification
Regression
Clustering
Pipeline
Predicting the chances of infant survival with ML
Loading the data
Creating transformers
Creating an estimator
Creating a pipeline
Fitting the model
Evaluating the performance of the model
Saving the model
Parameter hyper-tuning
Grid search
Train-validation splitting
Other features of PySpark ML in action
Feature extraction
NLP - related feature extractors
Discretizing continuous variables
Standardizing continuous variables
Classification
Clustering
Finding clusters in the births dataset
Topic mining
Regression
Summary
7. GraphFrames
Introducing GraphFrames
Installing GraphFrames
Creating a library
Preparing your flights dataset
Building the graph
Executing simple queries
Determining the number of airports and trips
Determining the longest delay in this dataset
Determining the number of delayed versus on-time/early flights
What flights departing Seattle are most likely to have significant delays?
What states tend to have significant delays departing from Seattle?
Understanding vertex degrees
Determining the top transfer airports
Understanding motifs
Determining airport ranking using PageRank
Determining the most popular non-stop flights
Using Breadth-First Search
Visualizing flights using D3
Summary
8. TensorFrames
What is Deep Learning?
The need for neural networks and Deep Learning
What is feature engineering?
Bridging the data and algorithm
What is TensorFlow?
Installing Pip
Installing TensorFlow
Matrix multiplication using constants
Matrix multiplication using placeholders
Running the model
Running another model
Discussion
Introducing TensorFrames
TensorFrames – quick start
Configuration and setup
Launching a Spark cluster
Creating a TensorFrames library
Installing TensorFlow on your cluster
Using TensorFlow to add a constant to an existing column
Executing the Tensor graph
Blockwise reducing operations example
Building a DataFrame of vectors
Analysing the DataFrame
Computing elementwise sum and min of all vectors
Summary
9. Polyglot Persistence with Blaze
Installing Blaze
Polyglot persistence
Abstracting data
Working with NumPy arrays
Working with pandas' DataFrame
Working with files
Working with databases
Interacting with relational databases
Interacting with the MongoDB database
Data operations
Accessing columns
Symbolic transformations
Operations on columns
Reducing data
Joins
Summary
10. Structured Streaming
What is Spark Streaming?
Why do we need Spark Streaming?
What is the Spark Streaming application data flow?
Simple streaming application using DStreams
A quick primer on global aggregations
Introducing Structured Streaming
Summary
11. Packaging Spark Applications
The spark-submit command
Command line parameters
Deploying the app programmatically
Configuring your SparkSession
Creating SparkSession
Modularizing code
Structure of the module
Calculating the distance between two points
Converting distance units
Building an egg
User defined functions in Spark
Submitting a job
Monitoring execution
Databricks Jobs
Summary
Index
Learning PySpark
Learning PySpark
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: February 2017
Production reference: 1220217
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78646-370-8
www.packtpub.com
Credits
Authors
Tomasz Drabas
Denny Lee
Reviewer
Holden Karau
Commissioning Editor
Amey Varangaonkar
Acquisition Editor
Prachi Bisht
Content Development Editor
Amrita Noronha
Technical Editor
Akash Patel
Copy Editor
Safis Editing
Project Coordinator
Shweta H Birwatkar
Proofreader
Safis Editing
Indexer
Aishwarya Gangawane
Graphics
Disha Haria
Production Coordinator
Aparna Bhagat
Cover Work
Aparna Bhagat
Foreword
Thank you for choosing this book to start your PySpark adventures, I hope you are as excited as I am. When Denny Lee first told me about this new book I was delighted-one of the most important things that makes Apache Spark such a wonderful platform, is supporting both the Java/Scala/JVM worlds and Python (and more recently R) worlds. Many of the previous books for Spark have been focused on either all of the core languages, or primarily focused on JVM languages, so it's great to see PySpark get its chance to shine with a dedicated book from such experienced Spark educators. By supporting both of these different worlds, we are able to more effectively work together as Data Scientists and Data Engineers, while stealing the best ideas from each other's communities.
It has been a privilege to have the opportunity to review early versions of this book, which has only increased my excitement for the project. I've had the privilege of being at some of the same conferences and meetups and watching the authors introduce new concepts in the world of Spark to a variety of audiences (from first timers to old hands), and they've done a great job distilling their experience for this book. The experience of the authors shines through with everything from their explanations to the topics covered. Beyond simply introducing PySpark they have also taken the time to look at up and coming packages from the community, such as GraphFrames and TensorFrames.
I think the community is one of those often-overlooked components when deciding what tools to use, and Python has a great community and I'm looking forward to you joining the Python Spark community. So, enjoy your adventure; I know you are in good hands with Denny Lee and Tomek Drabas. I truly believe that by having a diverse community of Spark users we will be able to make better tools useful for everyone, so I hope to see you around at one of the conferences, meetups, or mailing lists soon :)
Holden Karau
P.S.
I owe Denny a beer; if you want to buy him a Bud Light lime (or lime-a-rita) for me I'd be much obliged (although he might not be quite as amused as I am).
About the Authors
Tomasz Drabas is a Data Scientist working for Microsoft and currently residing in the Seattle area. He has over 13 years of experience in data analytics and data science in numerous fields: advanced technology, airlines, telecommunications, finance, and consulting he gained while working on three continents: Europe, Australia, and North America. While in Australia, Tomasz has been working on his PhD in Operations Research with a focus on choice modeling and revenue management applications in the airline industry.
At Microsoft, Tomasz works with big data on a daily basis, solving machine learning problems such as anomaly detection, churn prediction, and pattern recognition using Spark.
Tomasz has also authored the Practical Data Analysis Cookbook published by Packt Publishing in 2016.
I would like to thank my family: Rachel, Skye, and Albert—you are the love of my life and I cherish every day I spend with you! Thank you for always standing by me and for encouraging me to push my career goals further and further. Also, to my family and my in-laws for putting up with me (in general).
There are many more people that have influenced me over the years that I would have to write another book to thank them all. You know who you are and I want to thank you from the bottom of my heart!
However, I would not have gotten through my PhD if it was not for Czesia Wieruszewska; Czesiu - dziękuję za Twoją pomoc bez której nie rozpocząłbym mojej podróży po Antypodach. Along with Krzys Krzysztoszek, you guys have always believed in me! Thank you!
Denny Lee is a Principal Program Manager at Microsoft for the Azure DocumentDB team—Microsoft's blazing fast, planet-scale managed document store service. He is a hands-on distributed systems and data science engineer with more than 18 years of experience developing Internet-scale infrastructure, data platforms, and predictive analytics systems for both on-premise and cloud environments.
He has extensive experience of building greenfield teams as well as turnaround/change catalyst. Prior to joining the Azure DocumentDB team, Denny worked as a Technology Evangelist at Databricks; he has been working with Apache Spark since 0.5. He was also the Senior Director of Data Sciences Engineering at Concur, and was on the incubation team that built Microsoft's Hadoop on Windows and Azure service (currently known as HDInsight). Denny also has a Masters in Biomedical Informatics from Oregon Health and Sciences University and has architected and implemented powerful data solutions for enterprise healthcare customers for the last 15 years.
I would like to thank my wonderful spouse, Hua-Ping, and my awesome daughters, Isabella and Samantha. You are the ones who keep me grounded and help me reach for the stars!
About the Reviewer
Holden Karau is transgender Canadian, and an active open source contributor. When not in San Francisco working as a software development engineer at IBM's Spark Technology Center, Holden talks internationally on Spark and holds office hours at coffee shops at home and abroad. Holden is a co-author of numerous books on Spark including High Performance Spark (which she believes is the gift of the season for those with expense accounts) & Learning Spark. Holden is a Spark committer, specializing in PySpark and Machine Learning. Prior to IBM she worked on a variety of distributed, search, and classification problems at Alpine, Databricks, Google, Foursquare, and Amazon. She graduated from the University of Waterloo with a Bachelor of Mathematics in Computer Science. Outside of software she enjoys playing with fire, welding, scooters, poutine, and dancing.
www.PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1786463709.
If you'd like to join our team of regular reviewers, you can email us at <[email protected]>. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Preface
It is estimated that in 2013 the whole world produced around 4.4 zettabytes of data; that is, 4.4 billion terabytes! By 2020, we (as the human race) are expected to produce ten times that. With data getting larger literally by the second, and given the growing appetite for making sense out of it, in 2004 Google employees Jeffrey Dean and Sanjay Ghemawat published the seminal paper MapReduce: Simplified Data Processing on Large Clusters. Since then, technologies leveraging the concept started growing very quickly with Apache Hadoop initially being the most popular. It ultimately created a Hadoop ecosystem that included abstraction layers such as Pig, Hive, and Mahout – all leveraging this simple concept of map and reduce.
However, even though capable of chewing through petabytes of data daily, MapReduce is a fairly restricted programming framework. Also, most of the tasks require reading and writing to disk. Seeing these drawbacks, in 2009 Matei Zaharia started working on Spark as part of his PhD. Spark was first released in 2012. Even though Spark is based on the same MapReduce concept, its advanced ways of dealing with data and organizing tasks make it 100x faster than Hadoop (for in-memory computations).
In this book, we will guide you through the latest incarnation of Apache Spark using Python. We will show you how to read structured and unstructured data, how to use some fundamental data types available in PySpark, build machine learning models, operate on graphs, read streaming data, and deploy your models in the cloud. Each chapter will tackle different problem, and by the end of the book we hope you will be knowledgeable enough to solve other problems we did not have space to cover here.
What this book covers
Chapter 1, Understanding Spark, provides an introduction into the Spark world with an overview of the technology and the jobs organization concepts.
Chapter 2, Resilient Distributed Datasets, covers RDDs, the fundamental, schema-less data structure available in PySpark.
Chapter 3, DataFrames, provides a detailed overview of a data structure that bridges the gap between Scala and Python in terms of efficiency.
Chapter 4, Prepare Data for Modeling, guides the reader through the process of cleaning up and transforming data in the Spark environment.
Chapter 5, Introducing MLlib, introduces the machine learning library that works on RDDs and reviews the most useful machine learning models.
Chapter 6, Introducing the ML Package, covers the current mainstream machine learning library and provides an overview of all the models currently available.
Chapter 7, GraphFrames, will guide you through the new structure that makes solving problems with graphs easy.
Chapter 8, TensorFrames, introduces the bridge between Spark and the Deep Learning world of TensorFlow.
Chapter 9, Polyglot Persistence with Blaze, describes how Blaze can be paired with Spark for even easier abstraction of data from various sources.
Chapter 10, Structured Streaming, provides an overview of streaming tools available in PySpark.
Chapter 11, Packaging Spark Applications, will guide you through the steps of modularizing your code and submitting it for execution to Spark through command-line interface.
For more information, we have provided two bonus chapters as follows:
Installing Spark: https://www.packtpub.com/sites/default/files/downloads/InstallingSpark.pdf
Free Spark Cloud Offering: https://www.packtpub.com/sites/default/files/downloads/FreeSparkCloudOffering.pdf
What you need for this book
For this book you need a personal computer (can be either Windows machine, Mac, or Linux). To run Apache Spark, you will need Java 7+ and an installed and configured Python 2.6+ or 3.4+ environment; we use the Anaconda distribution of Python in version 3.5, which can be downloaded from https://www.continuum.io/downloads.
The Python modules we randomly use throughout the book come preinstalled with Anaconda. We also use GraphFrames and TensorFrames that can be loaded dynamically while starting a Spark instance: to load these you just need an Internet connection. It is fine if some of those modules are not currently installed on your machine – we will guide you through the installation process.
Who this book is for
This book is for everyone who wants to learn the fastest-growing technology in big data: Apache Spark. We hope that even the more advanced practitioners from the field of data science can find some of the examples refreshing and the more advanced topics interesting.
Conventions
In this book, you will find a number of styles of text