Exploring Hadoop Ecosystem (Volume 2): Stream Processing
By Wei Liu
()
About this ebook
Wei Liu
Wei Liu is Doctor of engineering at Beijing University of Aeronautics and Astronautics, Professor of Beijing University of Posts and Telecommunications, Visiting scholar of Cambridge University, Expert of Artificial Intelligence Group, Center for strategy and security, Tsinghua University and vice chairman of cognitive branch of the China Association of Command-and-Control His research interests include human-computer integration intelligence, cognitive engineering, human-machine- environment system engineering, future situation awareness mode and behavior analysis / prediction technology, etc. So far, he has published more than 70 papers, 4 monographs and 2 translations. At present, he is a distinguished expert of Expert Committee of China information and Electronic Engineering Science and technology development center, an appraisal expert of National Natural Science Foundation of China, a member of national ergonomics Standardization Technical Committee, and a senior member of the Chinese artificial intelligence society.
Read more from Wei Liu
Introduction to Hybrid Vehicle System Modeling and Control Rating: 4 out of 5 stars4/5Exploring Hadoop Ecosystem (Volume 1): Batch Processing Rating: 0 out of 5 stars0 ratingsHybrid Electric Vehicle System Modeling and Control Rating: 0 out of 5 stars0 ratingsLow-cost Smart Antennas Rating: 0 out of 5 stars0 ratingsHandbook of Thoracic Malignancies and Esophageal Related Cancer Rating: 0 out of 5 stars0 ratings
Related to Exploring Hadoop Ecosystem (Volume 2)
Related ebooks
Learn Hadoop in 24 Hours Rating: 0 out of 5 stars0 ratingsLearn Cassandra in 24 Hours Rating: 0 out of 5 stars0 ratingsLearning Hadoop 2 Rating: 4 out of 5 stars4/5Mastering Apache Cassandra - Second Edition Rating: 0 out of 5 stars0 ratingsGoogle Cloud Platform for Data Engineering: From Beginner to Data Engineer using Google Cloud Platform Rating: 5 out of 5 stars5/5HDInsight Essentials - Second Edition Rating: 0 out of 5 stars0 ratingsKafka Up and Running for Network DevOps: Set Your Network Data in Motion Rating: 0 out of 5 stars0 ratingsLearning ELK Stack Rating: 0 out of 5 stars0 ratingsLearn Hive in 24 Hours Rating: 0 out of 5 stars0 ratingsHadoop Real-World Solutions Cookbook - Second Edition Rating: 0 out of 5 stars0 ratingsMonitoring Hadoop Rating: 0 out of 5 stars0 ratingsHadoop BIG DATA Interview Questions You'll Most Likely Be Asked Rating: 0 out of 5 stars0 ratingsHadoop in Practice Rating: 0 out of 5 stars0 ratingsBig Data and Hadoop: Learn by example Rating: 0 out of 5 stars0 ratingsOptimizing Hadoop for MapReduce Rating: 0 out of 5 stars0 ratingsLearning Azure DocumentDB Rating: 0 out of 5 stars0 ratingsNginx Troubleshooting Rating: 0 out of 5 stars0 ratingsGetting Started with Big Data Query using Apache Impala Rating: 0 out of 5 stars0 ratingsThe Data Science Workshop: A New, Interactive Approach to Learning Data Science Rating: 0 out of 5 stars0 ratingsInstant Pentaho Data Integration Kitchen Rating: 0 out of 5 stars0 ratingsPostgreSQL 9 High Availability Cookbook Rating: 5 out of 5 stars5/5Apache Spark 2.x Cookbook Rating: 0 out of 5 stars0 ratingsFast Data Processing with Spark 2 - Third Edition Rating: 0 out of 5 stars0 ratingsMariaDB Complete Self-Assessment Guide Rating: 0 out of 5 stars0 ratingsHadoop Essentials Rating: 5 out of 5 stars5/5Data architect A Complete Guide - 2019 Edition Rating: 0 out of 5 stars0 ratingsHadoop: Data Processing and Modelling Rating: 0 out of 5 stars0 ratingsNiFi A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratings
Computers For You
The Invisible Rainbow: A History of Electricity and Life Rating: 5 out of 5 stars5/5Procreate for Beginners: Introduction to Procreate for Drawing and Illustrating on the iPad Rating: 0 out of 5 stars0 ratingsSlenderman: Online Obsession, Mental Illness, and the Violent Crime of Two Midwestern Girls Rating: 4 out of 5 stars4/5The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution Rating: 4 out of 5 stars4/5Elon Musk Rating: 4 out of 5 stars4/5Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics Rating: 4 out of 5 stars4/5CompTIA IT Fundamentals (ITF+) Study Guide: Exam FC0-U61 Rating: 0 out of 5 stars0 ratingsMastering ChatGPT: 21 Prompts Templates for Effortless Writing Rating: 5 out of 5 stars5/5101 Awesome Builds: Minecraft® Secrets from the World's Greatest Crafters Rating: 4 out of 5 stars4/5The Professional Voiceover Handbook: Voiceover training, #1 Rating: 5 out of 5 stars5/5The Complete Powershell Training for Beginners Rating: 0 out of 5 stars0 ratingsEverybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Rating: 4 out of 5 stars4/5The ChatGPT Millionaire Handbook: Make Money Online With the Power of AI Technology Rating: 4 out of 5 stars4/5The Hacker Crackdown: Law and Disorder on the Electronic Frontier Rating: 4 out of 5 stars4/5Alan Turing: The Enigma: The Book That Inspired the Film The Imitation Game - Updated Edition Rating: 4 out of 5 stars4/5CompTIA Security+ Get Certified Get Ahead: SY0-701 Study Guide Rating: 5 out of 5 stars5/5Uncanny Valley: A Memoir Rating: 4 out of 5 stars4/5How to Create Cpn Numbers the Right way: A Step by Step Guide to Creating cpn Numbers Legally Rating: 4 out of 5 stars4/5Ultimate Guide to Mastering Command Blocks!: Minecraft Keys to Unlocking Secret Commands Rating: 5 out of 5 stars5/5CompTIA Security+ Practice Questions Rating: 2 out of 5 stars2/5Deep Search: How to Explore the Internet More Effectively Rating: 5 out of 5 stars5/5The Huffington Post Complete Guide to Blogging Rating: 3 out of 5 stars3/5
Reviews for Exploring Hadoop Ecosystem (Volume 2)
0 ratings0 reviews
Book preview
Exploring Hadoop Ecosystem (Volume 2) - Wei Liu
Exploring Hadoop Ecosystem (Volume 2)
Stream Processing
(Spark Core, Spark SQL, Spark Streaming, Spark Structured Streaming, Kafka)
Wei Liu
Exploring Hadoop Ecosystem (volume 2) Stream Processing
by Wei Liu
Copyright © 2020 by Wei Liu
All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, contact [email protected].
ISBN: 978-1-6671-8450-0
First Edition
For the future
ABOUT THE AUTHOR
Wei Liu, an enterprise architect focusing on the big data, graduated from college in 2002, has 18 years of experience in the software development and architecture. He now lives in Beijing with his family. His email: [email protected].
TABLE OF CONTENTS
Batch processing vs Stream processing
Scala
Overview
sbt
Development Environment
Values and Variables
Basic Types
Inheritance Hierarchy
Conditional Expressions
Block Expressions
Loops
Collections
Tuple
Class and Object
Trait
Functional Programming
Currying
Type Parameters
Type Checks and Casts
Pattern Matching
Implicits
try/catch/finally
Packages
Chapter 1. Spark
Overview
Spark Architecture
SparkSession
Spark on YARN
Deploying Applications
Interactive Shells
Building Applications
RDD
Partitioning
DAG
RDD Dependency
Spark Execution Model
DAGScheduler
TaskScheduler
Data Locality
Task Execution
Shuffle
Memory Management
Blocks
RDD Persistence
RDD Checkpointing
Shared Variables
Chapter 2. Spark SQL
Shark
Spark SQL
Spark SQL Thrift Server
Spark SQL CLI
Dataset and DataFrame
Catalyst
Data Source APIs
Metastore
Hive Integration
Local Development Environment
Spark UI
Join Types
Join Implementation
Performance Tuning
Chapter 3. Spark Streaming
DStream
DStreamGraph
JobScheduler
Receiver-Based Approach
Direct Approach
Checkpointing
Transformations
Window Operations
Fault Tolerance
Kafka Exactly-once Semantics
Chapter 4. Spark Structured Streaming
Programming Model
Fault Tolerance Semantics
Word Count Example
State Management
Event Time Processing
Operations
Window Operations on Event Time
Watermarking
Deduplication
Join
Kafka Integration
Micro-batch vs Continuous
Chapter 5. Kafka
Message
Schema
Topic and Partitions
Log Flush
Log Cleanup
Replication
CLI Tools
Kafka Cluster
Producer
Consumer
Message Delivery Semantics
Java Application
Kafka Connect
Mirroring
UI Tools
Data Lake Architecture
Lambda Architecture
Kappa Architecture
Batch processing vs Stream processing
Today we are analyzing Terabytes and Petabytes of data in the Hadoop Ecosystem. There are generally two ways of processing data today- Batch Processing and Stream Processing. The distinction between batch processing and stream processing is one of the most fundamental principles in the big data world.
Batch Processing
Batch processing is the processing of a large volume of data all at once. It is most often used when dealing with very large amounts of data, or when data sources are legacy systems that are not capable of delivering data in streams.
Big data solutions often use long-running batch jobs to filter, aggregate, and prepare the data for analysis. Usually these jobs involve reading source files from scalable storage (like HDFS), processing them, and writing the output to new files in scalable storage. The key requirement of such batch processing engines is the ability to scale out computations, in order to handle a large volume of data. Unlike stream processing, batch processing is expected to have latencies that measure in minutes to hours.
A batch processing architecture has the following logical components:
Data storage
Typically, a distributed file store that can serve as a repository for high volumes of large files in various formats. Generically, this kind of store is often referred to as a data lake.
Batch processing
The high-volume nature of big data often means that solutions must process data files using long-running batch jobs to filter, aggregate, and otherwise prepare the data for analysis. Usually, these jobs involve reading source files, processing them, and writing the output to new files.
Analytical data store
Many big data solutions are designed to prepare data for analysis and then serve the processed data in a structured format that can be queried using analytical tools.
Analysis and reporting
The goal of most big data solutions is to provide insights into the data through analysis and reporting.
Orchestration
With batch processing, typically some orchestration is required to migrate or copy the data into our data storage, batch processing, analytical data store, and reporting layers.
Stream Processing
Stream processing is defined as the processing of unbounded stream of input data, with very short latency requirements for processing- measured in milliseconds or seconds. The incoming data typically arrives in an unstructured or semi-structured format, such as JSON, and has the same processing requirements as batch processing, but with shorter turnaround times to support real-time consumption. Processed data is often written to an analytical data store, which is optimized for analytics and visualization. The processed data can also be ingested directly into the analytics and reporting layer for analysis, business intelligence, and real-time dashboard visualization.
While batch processing handles a large batch of data, stream processing handles individual records or micro batches of few records. By building data streams, we can feed data into analytics tools as soon as it is generated and get near-instant analytics results using platforms like Spark Streaming.
A stream processing architecture has the following logical components:
Streaming message ingestion
The architecture must include a way to capture and store streaming messages to be consumed by a stream processing consumer. In simple cases, this could be implemented as a simple data store in which new messages are deposited in a folder. But often the solution requires a message broker, such as Kafka, that acts as a buffer for the messages. The message broker should support scale-out processing and reliable delivery.
Stream processing
After capturing streaming messages, the solution must process them by filtering, aggregating, and preparing the data for analysis.
Analytical data store
Many big data solutions are designed to prepare data for analysis and then serve the processed data in a structured format that can be queried using analytical tools.
Analysis and reporting
The goal of most big data solutions is to provide insights into the data through analysis and reporting.
Stream processing refers to a method of continuous computation that happens as data is flowing through the system. There are no compulsory time limitations in stream processing. The terms stream processing and real-time processing are often used interchangeably. But stream processing is not a synonym for real-time processing. The term real-time has no precise industry definition. The meaning of real-time varies significantly between businesses. Some use cases, such as online fraud detection, may require processing to complete within milliseconds, but for others multiple seconds or even minutes might be sufficiently fast. Usually, a system is called a real-time system if it has tight deadlines within which a result is guaranteed. In practice real-time systems are extremely hard to implement using common software systems. When we use the term real-time, we mean systems that can respond fast. Here fast can be milliseconds to seconds.
Scala
Overview
Scala, stands for scalable language, was created by Martin Odersky and was first released in 2003. The language is so named because it was designed to grow with the demands of its users. We can apply Scala to a wide range of programming tasks, from writing small scripts to building large systems.
Scala is compiled to class files that we package as JAR files, which is executed by the Java Virtual Machine (JVM). JVM is a cross platform runtime engine that can execute instructions compiled into Java bytecode. Scala can be compiled into bytecode and runs on JVM. This means that Scala and Java have a common run-time platform. Scala interoperates seamlessly with all Java libraries. Scala enables us to use all the classes of the Java SDK's, and also our custom Java classes, or Java open source projects.
But Scala doesn't just target the Java Virtual Machine. Scala.js is a compiler that compiles Scala source code to equivalent JavaScript code. That lets us write Scala code that we can run in a web browser, or other environments (Chrome plugins, Node.js, etc.) where JavaScript is supported. This enables us to write both the server-side and client-side code of web applications in only one language.
Scala is a multi-paradigm programming language and supports functional as well as object oriented paradigms. Scala is a pure object-oriented language in the sense that every value is an object. Scala is also a functional language in the sense that every function is a value.
Scala is strongly statically typed.
Statically vs Dynamically
Type Checking is the process of verifying and enforcing the constraints of types. Type Checking may occur either at compile-time or at run-time. Many Programming languages throw type errors which halts the run-time or compilation of the program, depending on the language type- statically or dynamically typed.
A language is statically-typed if the type of a variable is known at compile-time instead of at run-time. A language is dynamically-typed if the type of a variable is checked during run-time.
Strongly vs Weakly
A strongly-typed language is one in which variables are bound to specific data types, and will result in type errors if types do not match up as expected in the expression, regardless of when type checking occurs.
A weakly-typed language is a language in which variables are not bound to a specific data type; they still have a type, but type safety constraints are lower compared to strongly-typed languages.
classification of some programming languages
sbt
We can use several different tools to build our Scala projects, including Ant, Maven, Gradle, and more. But sbt was the first build tool that was specifically created for Scala and is most commonly used build tool in the Scala community. It's similar to Java Maven or .net NuGet.
sbt (Scala Build Tool) is a highly interactive build tool, which can be used for Scala, Java, and more. It requires Java 8 or later. It provides a parallel execution engine and configuration system that allow us design an efficient and robust script to build our software.
Build Definition
A build definition defines a set of subprojects. Each subproject holds a sequence of key-value pairs called setting expressions using build.sbt DSL. The build.sbt DSL is a domain-specific language used to construct a DAG of settings.
A setting expression consists of three parts: key, operator, body. For example,
A key is an instance of SettingKey[T], TaskKey[T], or InputKey[T] where T is the expected value type.
A SettingKey represents a setting, a TaskKey represents a task, and a InputKey represents an input task. A given key always refers to either a task or a setting. A setting in sbt is just a value. It could be the name of the project, or the version of Scala to use. A task is an operation such as compile or package. It may return Unit (Unit is void for Scala), or it may return a value related to the task, for example package is a TaskKey[File] and its value is the jar file it creates. An input task parses user input and produce a task to run.
Built-in Keys vs Custom Keys
The built-in keys are just fields in an object called Keys. A build.sbt implicitly has an import sbt.Keys._, so sbt.Keys.name can be referred to as name.
Each type of keys can be defined with their respective creation methods: settingKey, taskKey, and inputKey. Each method expects the type of the value associated with the key as well as a description. For example, define a key for a new task called hello,
lazy val hello = taskKey[Unit](An example task
)
Defining settings and tasks
sbt consists of two things, settings and tasks. Both settings and tasks produce values, but there are two major differences between them.
An example of setting
lazy val root = (project in file(.
))
.settings(
name := hello
)
An example of task
By default, sbt runs all of the tasks in parallel, but using the dependency tree it can work out what should be sequential and what can be parallel. A task in sbt is Scala code. There isn't any intermediate XML, we just write the code directly in our build configuration.
// first define a task key
lazy val hello = taskKey[Unit](An example task
)
// then implement the task key
lazy val root = (project in file(.
))
.settings(
hello := { println(Hello!
) }
)
An example of input task
An input task is any task that can accept additional user input before execution.
val demo = inputKey[Unit](A demo input task.
)
lazy val root = (project in file(.
))
.settings(
demo := {
// get the result of parsing
val args: Seq[String] = spaceDelimited(
).parsed
// Here, we also use the value of the `scalaVersion` setting
println(The current Scala version is
+ scalaVersion.value)
println(The arguments to demo were:
)
args foreach println
}
)
Multi-project Builds
A subproject is defined by declaring a lazy val of type Project. For example,
lazy val util = (project in file(util
))
lazy val core = (project in file(core
))
We can keep multiple related subprojects in a single build definition. Each subproject in a build definition has its own source directories, generates its own jar file when we run package.
To factor out common settings across multiple projects, define the settings scoped to ThisBuild.
ThisBuild / organization := com.example
ThisBuild / version := 0.1.0-SNAPSHOT
ThisBuild / scalaVersion := 2.12.10
lazy val core = (project in file(core
))
.settings(
// other settings
)
lazy val util = (project in file(util
))
.settings(
// other settings
)
These settings, which are written directly into the build.sbt file instead of putting them inside a .settings(...) call, are called bare style.
Another way to factor out common settings across multiple projects is to create a sequence named commonSettings and call settings method on each project.
lazy val commonSettings = Seq(
target := { baseDirectory.value / target2
}
)
lazy val core = (project in file(core
))
.settings(
commonSettings,
// other settings
)
lazy val util = (project in file(util
))
.settings(
commonSettings,
// other settings
)
Setting up a build
Every project using sbt should have two files:
The build.properties file is used to inform sbt which version it should use for our build. While this file can be used to specify several things, it's commonly only used to specify the sbt version. The build.sbt file defines the actual build.
The project directory can contain .scala files that define helper objects and one-off plugins. These .scala files under project directory are part of the build definition. We can create project/Dependencies.scala to track dependencies in one place.
import sbt._
object Dependencies {
lazy val scalaTest = org.scalatest
%% scalatest
% 3.0.8
}
This Dependencies object will be available in build.sbt. To make it easier to use the val defined in Dependencies.scala, add import Dependencies._ in build.sbt file.
We can write any Scala code in .scala files, including top-level classes and objects. The recommended approach is to define most settings in a Multi-project build.sbt file, and using project/*.scala files for task implementations or to share values, such as keys.
The project directory is one special directory in an sbt build. It contains definitions which apply to the build itself, and not the artifact that sbt is building. When sbt is trying to construct our project definition, it reads the .sbt and .scala files files in project directory, which forms the build definition for the build project itself. It then uses this to help read the .sbt files in the base directory. It compiles everything that it finds and this gives us our build definition. sbt then runs this build and creates our artifact, jar or whatever.
Note that the project directory is itself a recursive structure.
Development Environment
The most popular way to work in Scala is using Scala through sbt within an IDE.
Using sbt on Command Line
First we need to install brew, if we don't have it yet.
/bin/bash -c $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)
Then run this command to install sbt.
brew install sbt
Following the convention, here we setup a simple helloworld project. This command pulls the hello-world template from GitHub. When prompted, name the application helloworld. This will create a project called helloworld.
sbt new scala/scala-seed.g8
Then cd to the helloworld folder.
cd helloworld
The following command will open up the sbt console.
sbt
图片包含 文字 描述已自动生成Then type run. We will get the following output.
[info] Compiling 1 Scala source to /Projects/sbt/helloworld/target/scala-2.13/classes ...
[info] running example.Hello
hello
[success] Total time: 3 s, completed March 8, 2020 1:47:10 AM
We don't need to install Scala separately, sbt will download Scala for us.
sbt Directory structure
When we create the project, the default directory structure is shown in the left figure below. After we run sbt to open up the sbt console, two target/ folders are generated as shown in the right figure.
图片包含 屏幕截图, 文字 描述已自动生成图片包含 屏幕截图, 文字 描述已自动生成Base directory is the directory containing the project. Here helloworld/ is the base directory.
The build definition is described in build.sbt (actually any files named *.sbt) in the base directory. For example, project name, project version, scalaVersion, libraryDependencies, etc. Note that this file is written in Scala.
As build.sbt is written in Scala, we need to build build.sbt. This directory contains all the files needed for building build.sbt.
sbt uses the same directory structure as Maven for source files by default. Main contains anything going to production while test contains anything used for unit testing.
Generated files (compiled classes, packaged jars, managed files, caches, and documentation) will be written to the target directory by default.
Every subproject in SBT has a target directory. That's where its compiled artifacts go. In the project directory we can write scala code to implement build-related tasks and settings. These compiled artifacts go to the project/target directory.
Visual Studio Code
Install the Metals extension from the Visutal Studio Code Marketplace.
Then open a directory containing a build.sbt file. The extension activates when a .scala or .sbt file is opened.
The first time we open Metals in a new workspace it prompts us to import the build. Click Import build to start the installation step.
Once the import step completes, compilation starts for our open *.scala files.
Language Server Protocol
Metals is a Scala Language Server with rich IDE features.
The Language Server Protocol (LSP) defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc. The LSP was created by Microsoft to define a common language for programming language analyzers to speak.
Today, several companies have come together to support its growth, including Codenvy, Red Hat, and Sourcegraph, and the protocol is becoming supported by a rapidly growing list of editor and language communities.
Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature.
LSP creates the opportunity to reduce the m-times-n complexity problem of providing a high level of support for any programming language in any editor, IDE, or client to a simpler m-plus-n problem. For example, instead of the traditional practice of building a Python plugin for VSCode, a Python plugin for Sublime Text, a Python plugin for Vim, a Python plugin for Sourcegraph, and so on, for every language, LSP allows language communities to concentrate their efforts on a single, high performing language server that can provide code completion, hover tooltips, jump-to-definition, find-references, and more, while editor and client communities can concentrate on building a single, high performing, intuitive and idiomatic extension that can communicate with any language server to instantly provide deep language support.
LSP is a win for both language providers and tooling vendors.
Example of Language servers
Example of LSP clients
Scala REPL
The easiest way to get started with Scala is by using the Scala REPL, an interactive shell for writing Scala expressions and programs.
The Scala REPL is a command-line interpreter that we use as a playground area to test our Scala code. To start a REPL session, just type scala at our operating system command line. Behind the scenes, our input is quickly compiled into bytecode, and the bytecode is executed by JVM.
A read–eval–print loop (REPL), also termed