Frank Kane's Taming Big Data with Apache Spark and Python
By Frank Kane
()
About this ebook
- Understand how Spark can be distributed across computing clusters
- Develop and run Spark jobs efficiently using Python
- A hands-on tutorial by Frank Kane with over 15 real-world examples teaching you Big Data processing with Spark
If you are a data scientist or data analyst who wants to learn Big Data processing using Apache Spark and Python, this book is for you. If you have some programming experience in Python, and want to learn how to process large amounts of data using Apache Spark, Frank Kane’s Taming Big Data with Apache Spark and Python will also help you.
Frank Kane
Frank Kane (1912–1968) was the author of the Johnny Liddell mystery series, including Dead Weight, Trigger Mortis, Poisons Unknown, and many more.
Read more from Frank Kane
Poisons Unknown Rating: 4 out of 5 stars4/5Red Hot Ice Rating: 5 out of 5 stars5/5Bare Trap Rating: 5 out of 5 stars5/5Stacked Deck Rating: 5 out of 5 stars5/5Slay Ride Rating: 5 out of 5 stars5/5Green Light for Death Rating: 5 out of 5 stars5/5A Real Gone Guy Rating: 4 out of 5 stars4/5Johnny Liddell's Morgue Rating: 5 out of 5 stars5/5Time to Prey Rating: 5 out of 5 stars5/5A Short Bier Rating: 5 out of 5 stars5/5The Fatal Foursome Rating: 5 out of 5 stars5/5Black Cat Weekly #84 Rating: 0 out of 5 stars0 ratingsTrigger Mortis Rating: 4 out of 5 stars4/5Bare Trap: A Johnny Liddell Mystery Rating: 0 out of 5 stars0 ratingsBlack Cat Weekly #104 Rating: 0 out of 5 stars0 ratingsHands-On Data Science and Python Machine Learning: Perform data mining and machine learning efficiently using Python and Spark Rating: 0 out of 5 stars0 ratingsAbout Face: A Johnny Liddell Mystery Rating: 0 out of 5 stars0 ratingsBlack Cat Weekly #141 Rating: 0 out of 5 stars0 ratingsBullet Proof: A Johnny Liddell Mystery Rating: 0 out of 5 stars0 ratingsBlack Cat Weekly #66 Rating: 0 out of 5 stars0 ratingsBlack Cat Weekly #63 Rating: 0 out of 5 stars0 ratingsRed Hot Ice: A Johnny Liddell Mystery Rating: 0 out of 5 stars0 ratingsThe Icepick Artists Rating: 0 out of 5 stars0 ratingsDead Weight Rating: 0 out of 5 stars0 ratingsDead Weight: A Johnny Liddell Mystery Rating: 0 out of 5 stars0 ratingsBlack Cat Weekly #76 Rating: 0 out of 5 stars0 ratings
Related to Frank Kane's Taming Big Data with Apache Spark and Python
Related ebooks
Apache Spark 2.x Cookbook Rating: 0 out of 5 stars0 ratingsScala for Data Science Rating: 0 out of 5 stars0 ratingsPython Data Analysis - Second Edition Rating: 0 out of 5 stars0 ratingsMastering Python Design Patterns Rating: 0 out of 5 stars0 ratingsApache Spark Graph Processing Rating: 0 out of 5 stars0 ratingsReinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges Rating: 0 out of 5 stars0 ratingsLearning PySpark Rating: 0 out of 5 stars0 ratingsLearning Apache Spark 2 Rating: 0 out of 5 stars0 ratingsData Analysis with Python and PySpark Rating: 0 out of 5 stars0 ratingsMastering Machine Learning on AWS: Advanced machine learning in Python using SageMaker, Apache Spark, and TensorFlow Rating: 0 out of 5 stars0 ratingsOperationalizing Machine Learning Pipelines: Building Reusable and Reproducible Machine Learning Pipelines Using MLOps Rating: 0 out of 5 stars0 ratingsMachine Learning with Spark - Second Edition Rating: 0 out of 5 stars0 ratingsMastering Large Datasets with Python: Parallelize and Distribute Your Python Code Rating: 0 out of 5 stars0 ratingsData Lake Development with Big Data Rating: 0 out of 5 stars0 ratingsData Pipelines with Apache Airflow Rating: 0 out of 5 stars0 ratingsBig Data Analytics Rating: 0 out of 5 stars0 ratingsHadoop Real-World Solutions Cookbook - Second Edition Rating: 0 out of 5 stars0 ratingsThe Data Science Workshop: A New, Interactive Approach to Learning Data Science Rating: 0 out of 5 stars0 ratingsSpark in Action: Covers Apache Spark 3 with Examples in Java, Python, and Scala Rating: 0 out of 5 stars0 ratingsPractical Full Stack Machine Learning: A Guide to Build Reliable, Reusable, and Production-Ready Full Stack ML Solutions Rating: 0 out of 5 stars0 ratingsData Engineering on Azure Rating: 0 out of 5 stars0 ratingsMachine Learning Bookcamp: Build a portfolio of real-life projects Rating: 4 out of 5 stars4/5Machine Learning in Action Rating: 0 out of 5 stars0 ratingsMLOps Engineering at Scale Rating: 0 out of 5 stars0 ratingsMLOps A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsApache Spark for Data Science Cookbook Rating: 0 out of 5 stars0 ratingsGetting Started with Python Data Analysis Rating: 0 out of 5 stars0 ratings
Data Modeling & Design For You
Data Visualization: a successful design process Rating: 4 out of 5 stars4/5Spreadsheets To Cubes (Advanced Data Analytics for Small Medium Business): Data Science Rating: 0 out of 5 stars0 ratingsThinking in Algorithms: Strategic Thinking Skills, #2 Rating: 5 out of 5 stars5/5Mastering Agile User Stories Rating: 4 out of 5 stars4/5The Secrets of ChatGPT Prompt Engineering for Non-Developers Rating: 5 out of 5 stars5/5Data Analytics for Beginners: Introduction to Data Analytics Rating: 4 out of 5 stars4/5DAX Patterns: Second Edition Rating: 5 out of 5 stars5/5Neural Networks for Beginners: An Easy-to-Follow Introduction to Artificial Intelligence and Deep Learning Rating: 2 out of 5 stars2/5Tableau Cookbook – Recipes for Data Visualization Rating: 0 out of 5 stars0 ratingsLearning Social Media Analytics with R Rating: 0 out of 5 stars0 ratingsLiving in Data: A Citizen's Guide to a Better Information Future Rating: 4 out of 5 stars4/5Data Analytics with Python: Data Analytics in Python Using Pandas Rating: 3 out of 5 stars3/5Managing Data Using Excel Rating: 5 out of 5 stars5/5Raspberry Pi :Raspberry Pi Guide On Python & Projects Programming In Easy Steps Rating: 3 out of 5 stars3/5Learning Cypher Rating: 0 out of 5 stars0 ratingsWordPress For Beginners - How To Set Up A Self Hosted WordPress Blog Rating: 0 out of 5 stars0 ratingsLearn T-SQL Querying: A guide to developing efficient and elegant T-SQL code Rating: 0 out of 5 stars0 ratingsAdvanced Splunk Rating: 5 out of 5 stars5/5Supercharge Power BI: Power BI is Better When You Learn To Write DAX Rating: 5 out of 5 stars5/5Tailoring Prompts For Success - The Ultimate ChatGPT Prompt Engineering Guide Rating: 3 out of 5 stars3/5The Systems Thinker - Mental Models: The Systems Thinker Series, #3 Rating: 0 out of 5 stars0 ratingsAI and UX: Why Artificial Intelligence Needs User Experience Rating: 0 out of 5 stars0 ratingsAI-Driven Data Engineering Rating: 0 out of 5 stars0 ratingsPython Data Analysis Rating: 4 out of 5 stars4/5Neural Networks: Neural Networks Tools and Techniques for Beginners Rating: 5 out of 5 stars5/5
Reviews for Frank Kane's Taming Big Data with Apache Spark and Python
0 ratings0 reviews
Book preview
Frank Kane's Taming Big Data with Apache Spark and Python - Frank Kane
Frank Kane's Taming Big Data with Apache Spark and Python
         Â
Real-world examples to help you analyze large datasets with Apache Spark
         Â
Frank Kane
BIRMINGHAM - MUMBAI
< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN
http://www.w3.org/TR/REC-html40/loose.dtd
>
Frank Kane's Taming Big Data with Apache Spark and Python
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: June 2017
Production reference: 1290617
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78728-794-5
www.packtpub.com
Credits
About the Author
My name is Frank Kane. I spent nine years at amazon.com and imdb.com, wrangling millions of customer ratings and customer transactions to produce things such as personalized recommendations for movies and products and people who bought this also bought.
I tell you, I wish we had Apache Spark back then, when I spent years trying to solve these problems there. I hold 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, I left to start my own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787287947.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing Enthought Canopy
Installing the Java Development Kit
Installing Spark
Running Spark code
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Examining the ratings counter script
Running the ratings counter script
Summary
Spark Basics and Spark Examples
What is Spark?
Spark is scalable
Spark is fast
Spark is hot
Spark is not that hard
Components of Spark
Using Python with Spark
The Resilient Distributed Dataset (RDD)
What is the RDD?
The SparkContext object
Creating RDDs
Transforming RDDs
Map example
RDD actions
Ratings histogram walk-through
Understanding the code
Setting up the SparkContext object
Loading the data
Extract (MAP) the data we care about
Perform an action - count by value
Sort and display the results
Looking at the ratings-counter script in Canopy
Key/value RDDs and the average friends by age example
Key/value concepts - RDDs can hold key/value pairs
Creating a key/value RDD
What Spark can do with key/value data?
Mapping the values of a key/value RDD
The friends by age example
Parsing (mapping) the input data
Counting up the sum of friends and number of entries per age
Compute averages
Collect and display the results
Running the average friends by age example
Examining the script
Running the code
Filtering RDDs and the minimum temperature by location example
What is filter()
The source data for the minimum temperature by location example
Parse (map) the input data
Filter out all but the TMIN entries
Create (station ID, temperature) key/value pairs
Find minimum temperature by station ID
Collect and print results
Running the minimum temperature example and modifying it for maximums
Examining the min-temperatures script
Running the script
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Map versus flatmap
Map ()
Flatmap ()
Code sample - count the words in a book
Improving the word-count script with regular expressions
Text normalization
Examining the use of regular expressions in the word-count script
Running the code
Sorting the word count results
Step 1 - Implement countByValue() the hard way to create a new RDD
Step 2 - Sort the new RDD
Examining the script
Running the code
Find the total amount spent by customer
Introducing the problem
Strategy for solving the problem
Useful snippets of code
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
Advanced Examples of Spark Programs
Finding the most popular movie
Examining the popular-movies script
Getting results
Using broadcast variables to display movie names instead of ID numbers
Introducing broadcast variables
Examining the popular-movies-nicer.py script
Getting results
Finding the most popular superhero in a social graph
Superhero social networks
Input data format
Strategy
Running the script - discover who the most popular superhero is
Mapping input data to (hero ID, number of co-occurrences) per line
Adding up co-occurrence by hero ID
Flipping the (map) RDD to (number, hero ID)
Using max() and looking up the name of the winner
Getting results
Superhero degrees of separation - introducing the breadth-first search algorithm
Degrees of separation
How the breadth-first search algorithm works?
The initial condition of our social graph
First pass through the graph
Second pass through the graph
Third pass through the graph
Final pass through the graph
Accumulators and implementing BFS in Spark
Convert the input file into structured data
Writing code to convert Marvel-Graph.txt to BFS nodes
Iteratively process the RDD
Using a mapper and a reducer
How do we know when we're done?
Superhero degrees of separation - review the code and run it
Setting up an accumulator and using the convert to BFS function
Calling flatMap()
Calling an action
Calling reduceByKey
Getting results
Item-based collaborative filtering in Spark, cache(), and persist()
How does item-based collaborative filtering work?
Making item-based collaborative filtering a Spark problem
It's getting real
Caching RDDs
Running the similar-movies script using Spark's cluster manager
Examining the script
Getting results
Improving the quality of the similar movies example
Summary
Running Spark on a Cluster
Introducing Elastic MapReduce
Why use Elastic MapReduce?
Warning - Spark on EMR is not cheap
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Using .partitionBy()
Choosing a partition size
Creating similar movies from one million ratings - part 1
Changes to the script
Creating similar movies from one million ratings - part 2
Our strategy
Specifying memory per executor
Specifying a cluster manager
Running on a cluster
Setting up to run the movie-similarities-1m.py script on a cluster
Preparing the script
Creating a cluster
Connecting to the master node using SSH
Running the code
Creating similar movies from one million ratings – part 3
Assessing the results
Terminating the cluster
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Troubleshooting
Managing dependencies
Summary
SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Using SparkSQL in Python
More things you can do with DataFrames
Differences between DataFrames and DataSets
Shell access in SparkSQL
User-defined functions (UDFs)
Executing SQL commands and SQL-style functions on a DataFrame
Using SQL-style functions instead of queries
Using DataFrames instead of RDDs
Summary
Other Spark Technologies and Libraries
Introducing MLlib
MLlib capabilities
Special MLlib data types
For more information on machine learning
Making movie recommendations
Using MLlib to produce movie recommendations
Examining the movie-recommendations-als.py script
Analyzing the ALS recommendations results
Why did we get bad results?
Using DataFrames with MLlib
Examining the spark-linear-regression.py script
Getting results
Spark Streaming and GraphX
What is Spark Streaming?
GraphX
Summary
Where to Go From Here? – Learning More About Spark and Data Science
Preface
We will do some really quick housekeeping here, just so you know where to put all the stuff for this book. First, I want you to go to your hard drive, create a new folder called SparkCourse, and put it in a place where you're going to remember it is:
For me, I put that in my C drive in a folder called SparkCourse. This is where you're going to put everything for this book. As you go through the individual sections of this book, you'll see that there are resources provided for each one. There can be different kinds of resources, files, and downloads. When you download them, make sure you put them in this folder that you have created. This is the ultimate destination of everything you're going to download for this book, as you can see in my SparkCourse folder, shown in the following screenshot; you'll just accumulate all this stuff over time as you work your way through it:
So, remember where you put it all, you might need to refer to these files by their path, in this case, C:\SparkCourse. Just make sure you download them to a consistent place and you should be good to go. Also, be cognizant of the differences in file paths between operating systems. If you're on Mac or Linux, you're not going to have a C drive; you'll just have a slash and the full path name. Capitalization might be important, while it's not in Windows. Using forward slashes instead of backslashes in paths is another difference between other operating systems and Windows. So if you are using something other than Windows, just remember these differences, don't let them trip you up. If you see a path to a file and a script, make sure you adjust it accordingly to make sense of where you put these files and what your operating system is.
What this book covers
Chapter 1, Getting Started with Spark, covers basic installation instructions for Spark and its related software. This chapter illustrates a simple example of data analysis of real movie ratings data provided by different sets of people.
Chapter 2, Spark Basics and Simple Examples, provides a brief overview of what Spark is all about, who uses it, how it helps in analyzing big data, and why it is so popular.
Chapter3, Advanced Examples of Spark Programs, illustrates some advanced and complicated examples with Spark.
Chapter 4, Running Spark on a Cluster, talks about Spark Core, covering the things you can do with Spark, such as running Spark in the cloud on a cluster, analyzing a real cluster in the cloud using Spark, and so on.
Chapter 5, SparkSQL, DataFrames, and DataSets, introduces SparkSQL, which is an important concept of Spark, and explains how to deal with structured data formats using this.
Chapter 6, Other Spark Technologies and Libraries, talks about MLlib (Machine Learning library), which is very helpful if you want to work on data mining or machine learning-related jobs with Spark. This chapter also covers Spark Streaming and GraphX; technologies built on top of Spark.
Chapter 7, Where to Go From Here? - Learning More About Spark and Data Science, talks about some books related to Spark if the readers want to know more on this topic.
What you need for this book
For this book you’ll need a Python development environment (Python 3.5 or newer), a Canopy installer, Java Development Kit, and of course Spark itself (Spark 2.0 and beyond).
We'll show you how to install this software in first chapter of the book.
This book is based on the Windows operating system, so installations are provided according to it. If you have Mac or Linux, you can follow this URL http://media.sundog-soft.com/spark-python-install.pdf, which contains written instructions on getting everything set up on Mac OS and on Linux.
Who this book is for
I wrote this book for people who have at least some programming or scripting experience in their background. We're going to be using the Python programming language throughout this book, which is very easy to pick up, and I'm going to give you over 15 real hands-on examples of Spark Python scripts that you can run yourself, mess around with, and learn from. So, by the end of this book, you should have the skills needed to actually turn business problems into Spark problems, code up that Spark code on your own, and actually run it in the cluster on your own.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, path names, dummy URLs, user input, and Twitter handles are shown as follows: Now, you'll need to remember the path that we installed the JDK into, which in our case was C:\jdk.
A block of code is set as follows:
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
Any command-line input or output is written as follows:
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Now, if you're on Windows, I want you to right-click on the Enthought Canopy icon, go to Properties and then to Compatibility (this is on Windows 10), and make sure Run this program as an administrator is checked
Warnings or important notes appear in a box like this.
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the SUPPORT tab at the top.
Click on Code Downloads & Errata.
Enter the name of the book in the Search box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on Code Download.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Frank-Kanes-Taming-Big-Data-with-Apache-Spark-and-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Downloading the color images of this book
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/FrankKanesTamingBigDatawithApacheSparkandPython_ColorImages.pdf.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any