Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    Only $11.99/month after trial. Cancel anytime.

    Frank Kane's Taming Big Data with Apache Spark and Python
    Frank Kane's Taming Big Data with Apache Spark and Python
    Frank Kane's Taming Big Data with Apache Spark and Python
    Ebook452 pages3 hours

    Frank Kane's Taming Big Data with Apache Spark and Python

    By Frank Kane

    ()

    Read preview

    About this ebook

    About This Book
    • Understand how Spark can be distributed across computing clusters
    • Develop and run Spark jobs efficiently using Python
    • A hands-on tutorial by Frank Kane with over 15 real-world examples teaching you Big Data processing with Spark
    Who This Book Is For

    If you are a data scientist or data analyst who wants to learn Big Data processing using Apache Spark and Python, this book is for you. If you have some programming experience in Python, and want to learn how to process large amounts of data using Apache Spark, Frank Kane’s Taming Big Data with Apache Spark and Python will also help you.

    LanguageEnglish
    PublisherPackt Publishing
    Release dateJun 30, 2017
    ISBN9781787288300
    Frank Kane's Taming Big Data with Apache Spark and Python
    Author

    Frank Kane

    Frank Kane (1912–1968) was the author of the Johnny Liddell mystery series, including Dead Weight, Trigger Mortis, Poisons Unknown, and many more. 

    Read more from Frank Kane

    Related to Frank Kane's Taming Big Data with Apache Spark and Python

    Related ebooks

    Data Modeling & Design For You

    View More
    View More

    Related podcast episodes

    Related articles

    • Article

      Understanding ELT & ETL

      Apr 1, 2021

      Understanding ELT & ETL
      8 min read

    • Article

      The Future Of The Database

      Aug 27, 2019

      The Future Of The Database
      7 min read

    • Article

      Types Of Databases

      Aug 27, 2019

      NoSQL databases provide the performance, scalability and stability that’s required by the modern data-driven apps we interact with these days. But that is where the similarity between NoSQL systems end. In fact, it wouldn’t be wrong to say that the o

      1 min read

    • Article

      What is ELT?

      Apr 1, 2021

      It stands for extract, load, and transform- the processes a data pipeline uses for replicating the data from a source system into a target system such as a cloud data warehouse. 1. Extraction is the first step in which data is copied from the source

      6 min read

    • Article

      Tensor Flow 101

      Jan 27, 2020

      Tensor Flow 101
      4 min read

    • Article

      DJANGO Create A Database-driven Website

      Jun 4, 2019

      The Django web framework was named after the famous guitarist Django Reinhardt and was first created by web developers at a small newspaper in Kansas. The main goals of Django is to enable fast development of complex websites with database needs. It

      7 min read

    • Article

      AWS vs Azure

      Aug 22, 2023

      AWS vs Azure
      9 min read

    • Article

      How We Test

      May 9, 2024

      How We Test
      2 min read

    • Article

      How We Test

      Dec 7, 2023

      How We Test
      2 min read

    • Article

      How We Test

      Feb 8, 2024

      How We Test
      2 min read

    • Article

      How We Test

      Jan 4, 2024

      How We Test
      2 min read

    • Article

      Acer Nitro N50-610

      Oct 21, 2020

      DESKTOP PC | £799 from Currys www.snipca.com/36126 Brand of gold As a rule, you get a better desktop PC from a local (ie, British) company and a better laptop from an international brand. That’s because what matters with a PC is getting the best deal

      3 min read

    • Article

      How We Test

      Aug 13, 2020

      We run our own benchmarks on every Windows and macOS system we test. These are based around image editing, video editing and multitasking (where we run the video-editing benchmark while simultaneously playing back a 4K video). At the bottom of each l

      2 min read

    • Article

      How We Test

      Jan 7, 2021

      How We Test
      2 min read

    • Article

      Inside APC

      Feb 20, 2023

      APC is Australia’s oldest consumer technology magazine – having been consistently in print for over forty years, since our first issue way back in May 1980 – and we take that heritage and responsibility very seriously. While our focus is obviously on

      2 min read

    • Article

      Inside APC

      Feb 20, 2023

      APC is Australia’s oldest consumer technology magazine – having been consistently in print for over forty years, since our first issue way back in May 1980 – and we take that heritage and responsibility very seriously. While our focus is obviously on

      2 min read

    • Article

      Inside APC

      Jul 17, 2023

      Inside APC
      2 min read

    • Article

      How We Test

      Sep 10, 2020

      We run our own benchmarks on every Windows and macOS system we test. These are based around image editing, video editing and multitasking (where we run the video-editing benchmark while simultaneously playing back a 4K video). At the bottom of each l

      2 min read

    • Article

      Answers

      Mar 5, 2024

      Answers
      10 min read

    • Article

      Inside APC

      Jun 19, 2023

      APC is Australia’s oldest consumer technology magazine – having been consistently in print for over forty years, since our first issue way back in May 1980 – and we take that heritage and responsibility very seriously. While our focus is obviously on

      2 min read

    • Article

      Inside APC

      May 22, 2023

      Inside APC
      2 min read

    • Article

      Inside APC

      Apr 20, 2023

      APC is Australia’s oldest consumer technology magazine – having been consistently in print for over forty years, since our first issue way back in May 1980 – and we take that heritage and responsibility very seriously. While our focus is obviously on

      2 min read

    • Article

      Inside APC

      Mar 20, 2023

      APC is Australia’s oldest consumer technology magazine – having been consistently in print for over forty years, since our first issue way back in May 1980 – and we take that heritage and responsibility very seriously. While our focus is obviously on

      2 min read

    Related categories

    Reviews for Frank Kane's Taming Big Data with Apache Spark and Python

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Frank Kane's Taming Big Data with Apache Spark and Python - Frank Kane

      Frank Kane's Taming Big Data with Apache Spark and Python

                         

      Real-world examples to help you analyze large datasets with Apache Spark

                         

      Frank Kane

      BIRMINGHAM - MUMBAI

      < html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

      Frank Kane's Taming Big Data with Apache Spark and Python

      Copyright © 2017 Packt Publishing

      All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

      Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

      Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

      First published: June 2017

      Production reference: 1290617

      Published by Packt Publishing Ltd.

      Livery Place

      35 Livery Street

      Birmingham 

      B3 2PB, UK.

      ISBN 978-1-78728-794-5

      www.packtpub.com

      Credits

      About the Author

      My name is Frank Kane. I spent nine years at amazon.com and imdb.com, wrangling millions of customer ratings and customer transactions to produce things such as personalized recommendations for movies and products and people who bought this also bought. I tell you, I wish we had Apache Spark back then, when I spent years trying to solve these problems there. I hold 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, I left to start my own successful company, Sundog Software, which focuses on virtual reality environment technology, and teaching others about big data analysis.

      www.PacktPub.com

      For support files and downloads related to your book, please visit www.PacktPub.com.

      Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

      At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

      https://www.packtpub.com/mapt

      Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

      Why subscribe?

      Fully searchable across every book published by Packt

      Copy and paste, print, and bookmark content

      On demand and accessible via a web browser

      Customer Feedback

      Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787287947.

      If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

      Table of Contents

      Preface

      What this book covers

      What you need for this book

      Who this book is for

      Conventions

      Reader feedback

      Customer support

      Downloading the example code

      Downloading the color images of this book

      Errata

      Piracy

      Questions

      Getting Started with Spark

      Getting set up - installing Python, a JDK, and Spark and its dependencies

      Installing Enthought Canopy

      Installing the Java Development Kit

      Installing Spark

      Running Spark code

      Installing the MovieLens movie rating dataset

      Run your first Spark program - the ratings histogram example

      Examining the ratings counter script

      Running the ratings counter script

      Summary

      Spark Basics and Spark Examples

      What is Spark?

      Spark is scalable

      Spark is fast

      Spark is hot

      Spark is not that hard

      Components of Spark

      Using Python with Spark

      The Resilient Distributed Dataset (RDD)

      What is the RDD?

      The SparkContext object

      Creating RDDs

      Transforming RDDs

      Map example

      RDD actions

      Ratings histogram walk-through

      Understanding the code

      Setting up the SparkContext object

      Loading the data

      Extract (MAP) the data we care about

      Perform an action - count by value

      Sort and display the results

      Looking at the ratings-counter script in Canopy

      Key/value RDDs and the average friends by age example

      Key/value concepts - RDDs can hold key/value pairs

      Creating a key/value RDD

      What Spark can do with key/value data?

      Mapping the values of a key/value RDD

      The friends by age example

      Parsing (mapping) the input data

      Counting up the sum of friends and number of entries per age

      Compute averages

      Collect and display the results

      Running the average friends by age example

      Examining the script

      Running the code

      Filtering RDDs and the minimum temperature by location example

      What is filter()

      The source data for the minimum temperature by location example

      Parse (map) the input data

      Filter out all but the TMIN entries

      Create (station ID, temperature) key/value pairs

      Find minimum temperature by station ID

      Collect and print results

      Running the minimum temperature example and modifying it for maximums

      Examining the min-temperatures script

      Running the script

      Running the maximum temperature by location example

      Counting word occurrences using flatmap()

      Map versus flatmap

      Map ()

      Flatmap ()

      Code sample - count the words in a book

      Improving the word-count script with regular expressions

      Text normalization

      Examining the use of regular expressions in the word-count script

      Running the code

      Sorting the word count results

      Step 1 - Implement countByValue() the hard way to create a new RDD

      Step 2 - Sort the new RDD

      Examining the script

      Running the code

      Find the total amount spent by customer

      Introducing the problem

      Strategy for solving the problem

      Useful snippets of code

      Check your results and sort them by the total amount spent

      Check your sorted implementation and results against mine

      Summary

      Advanced Examples of Spark Programs

      Finding the most popular movie

      Examining the popular-movies script

      Getting results

      Using broadcast variables to display movie names instead of ID numbers

      Introducing broadcast variables

      Examining the popular-movies-nicer.py script

      Getting results

      Finding the most popular superhero in a social graph

      Superhero social networks

      Input data format

      Strategy

      Running the script - discover who the most popular superhero is

      Mapping input data to (hero ID, number of co-occurrences) per line

      Adding up co-occurrence by hero ID

      Flipping the (map) RDD to (number, hero ID)

      Using max() and looking up the name of the winner

      Getting results

      Superhero degrees of separation - introducing the breadth-first search algorithm

      Degrees of separation

      How the breadth-first search algorithm works?

      The initial condition of our social graph

      First pass through the graph

      Second pass through the graph

      Third pass through the graph

      Final pass through the graph

      Accumulators and implementing BFS in Spark

      Convert the input file into structured data

      Writing code to convert Marvel-Graph.txt to BFS nodes

      Iteratively process the RDD

      Using a mapper and a reducer

      How do we know when we're done?

      Superhero degrees of separation - review the code and run it

      Setting up an accumulator and using the convert to BFS function

      Calling flatMap()

      Calling an action

      Calling reduceByKey

      Getting results

      Item-based collaborative filtering in Spark, cache(), and persist()

      How does item-based collaborative filtering work?

      Making item-based collaborative filtering a Spark problem

      It's getting real

      Caching RDDs

      Running the similar-movies script using Spark's cluster manager

      Examining the script

      Getting results

      Improving the quality of the similar movies example

      Summary

      Running Spark on a Cluster

      Introducing Elastic MapReduce

      Why use Elastic MapReduce?

      Warning - Spark on EMR is not cheap

      Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY

      Partitioning

      Using .partitionBy()

      Choosing a partition size

      Creating similar movies from one million ratings - part 1

      Changes to the script

      Creating similar movies from one million ratings - part 2

      Our strategy

      Specifying memory per executor

      Specifying a cluster manager

      Running on a cluster

      Setting up to run the movie-similarities-1m.py script on a cluster

      Preparing the script

      Creating a cluster

      Connecting to the master node using SSH

      Running the code

      Creating similar movies from one million ratings – part 3

      Assessing the results

      Terminating the cluster

      Troubleshooting Spark on a cluster

      More troubleshooting and managing dependencies

      Troubleshooting

      Managing dependencies

      Summary

      SparkSQL, DataFrames, and DataSets

      Introducing SparkSQL

      Using SparkSQL in Python

      More things you can do with DataFrames

      Differences between DataFrames and DataSets

      Shell access in SparkSQL

      User-defined functions (UDFs)

      Executing SQL commands and SQL-style functions on a DataFrame

      Using SQL-style functions instead of queries

      Using DataFrames instead of RDDs

      Summary

      Other Spark Technologies and Libraries

      Introducing MLlib

      MLlib capabilities

      Special MLlib data types

      For more information on machine learning

      Making movie recommendations

      Using MLlib to produce movie recommendations

      Examining the movie-recommendations-als.py script

      Analyzing the ALS recommendations results

      Why did we get bad results?

      Using DataFrames with MLlib

      Examining the spark-linear-regression.py script

      Getting results

      Spark Streaming and GraphX

      What is Spark Streaming?

      GraphX

      Summary

      Where to Go From Here? – Learning More About Spark and Data Science

      Preface

      We will do some really quick housekeeping here, just so you know where to put all the stuff for this book. First, I want you to go to your hard drive, create a new folder called SparkCourse, and put it in a place where you're going to remember it is:

      For me, I put that in my C drive in a folder called SparkCourse. This is where you're going to put everything for this book. As you go through the individual sections of this book, you'll see that there are resources provided for each one. There can be different kinds of resources, files, and downloads. When you download them, make sure you put them in this folder that you have created. This is the ultimate destination of everything you're going to download for this book, as you can see in my SparkCourse folder, shown in the following screenshot; you'll just accumulate all this stuff over time as you work your way through it:

      So, remember where you put it all, you might need to refer to these files by their path, in this case, C:\SparkCourse. Just make sure you download them to a consistent place and you should be good to go. Also, be cognizant of the differences in file paths between operating systems. If you're on Mac or Linux, you're not going to have a C drive; you'll just have a slash and the full path name. Capitalization might be important, while it's not in Windows. Using forward slashes instead of backslashes in paths is another difference between other operating systems and Windows. So if you are using something other than Windows, just remember these differences, don't let them trip you up. If you see a path to a file and a script, make sure you adjust it accordingly to make sense of where you put these files and what your operating system is.

      What this book covers

      Chapter 1, Getting Started with Spark, covers basic installation instructions for Spark and its related software. This chapter illustrates a simple example of data analysis of real movie ratings data provided by different sets of people.

      Chapter 2, Spark Basics and Simple Examples, provides a brief overview of what Spark is all about, who uses it, how it helps in analyzing big data, and why it is so popular.

      Chapter3, Advanced Examples of Spark Programs, illustrates some advanced and complicated examples with Spark.

      Chapter 4, Running Spark on a Cluster, talks about Spark Core, covering the things you can do with Spark, such as running Spark in the cloud on a cluster, analyzing a real cluster in the cloud using Spark, and so on.

      Chapter 5, SparkSQL, DataFrames, and DataSets, introduces SparkSQL, which is an important concept of Spark, and explains how to deal with structured data formats using this.

      Chapter 6, Other Spark Technologies and Libraries, talks about MLlib (Machine Learning library), which is very helpful if you want to work on data mining or machine learning-related jobs with Spark. This chapter also covers Spark Streaming and GraphX; technologies built on top of Spark.

      Chapter 7, Where to Go From Here? - Learning More About Spark and Data Science, talks about some books related to Spark if the readers want to know more on this topic.

      What you need for this book

      For this book you’ll need a Python development environment (Python 3.5 or newer), a Canopy installer, Java Development Kit, and of course Spark itself (Spark 2.0 and beyond).

      We'll show you how to install this software in first chapter of the book.

      This book is based on the Windows operating system, so installations are provided according to it. If you have Mac or Linux, you can follow this URL http://media.sundog-soft.com/spark-python-install.pdf, which contains written instructions on getting everything set up on Mac OS and on Linux.

      Who this book is for

      I wrote this book for people who have at least some programming or scripting experience in their background. We're going to be using the Python programming language throughout this book, which is very easy to pick up, and I'm going to give you over 15 real hands-on examples of Spark Python scripts that you can run yourself, mess around with, and learn from. So, by the end of this book, you should have the skills needed to actually turn business problems into Spark problems, code up that Spark code on your own, and actually run it in the cluster on your own. 

      Conventions

      In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, path names, dummy URLs, user input, and Twitter handles are shown as follows: Now, you'll need to remember the path that we installed the JDK into, which in our case was C:\jdk. A block of code is set as follows:

      When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

      Any command-line input or output is written as follows:

      New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Now, if you're on Windows, I want you to right-click on the Enthought Canopy icon, go to Properties and then to Compatibility (this is on Windows 10), and make sure Run this program as an administrator is checked

      Warnings or important notes appear in a box like this.

      Tips and tricks appear like this.

      Reader feedback

      Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

      Customer support

      Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

      Downloading the example code

      You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:

      Log in or register to our website using your e-mail address and password.

      Hover the mouse pointer on the SUPPORT tab at the top.

      Click on Code Downloads & Errata.

      Enter the name of the book in the Search box.

      Select the book for which you're looking to download the code files.

      Choose from the drop-down menu where you purchased this book from.

      Click on Code Download.

      Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

      WinRAR / 7-Zip for Windows

      Zipeg / iZip / UnRarX for Mac

      7-Zip / PeaZip for Linux

      The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Frank-Kanes-Taming-Big-Data-with-Apache-Spark-and-Python. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

      Downloading the color images of this book

      We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/FrankKanesTamingBigDatawithApacheSparkandPython_ColorImages.pdf.

      Errata

      Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.

      Piracy

      Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.

      Questions

      If you have a problem with any

      Enjoying the preview?
      Page 1 of 1