Spark Streaming

Tathagata “TD” Das

Tathagata Das ( TD )
Whoami

•  Core committer on Apache Spark

•  Lead developer on Spark Streaming

•  On leave from PhD program in UC Berkeley

Big Data
Big Streaming Data
Big Streaming Data Processing
Fraud detection in bank transactions


Anomalies in sensor data


Cat videos in tweets

How to Process Big Streaming Data
Distributed
Processing System

Raw Data Streams Processed Data

Scales to hundreds of nodes

Achieves low latency

Efficiently recover from failures

Integrates with batch and interactive processing

What people have been doing?
> Build two stacks – one for batch, one for streaming
-  Often both process same data

> Existing frameworks cannot do both

-  Either, stream processing of 100s of MB/s with low latency
-  Or, batch processing of TBs of data with high latency

> Extremely painful to maintain two different stacks

-  Different programming models
-  Doubles implementation effort
-  Doubles operational effort
Fault-tolerant Stream Processing
> Traditional processing model
mutable  state  
–  Pipeline of nodes
input    
–  Each node maintains mutable state records  
–  Each input record updates the state node  1  
and new records are sent out
input     node  3  
records  
node  2  
> Mutable state is lost if node fails

> Making stateful stream processing

fault-tolerant is challenging!
Existing Streaming Systems

> Storm
-  Replays record if not processed by a node
-  Processes each record at least once
-  May update mutable state twice!
-  Mutable state can be lost due to failure!

> Trident – Use transactions to update state

-  Processes each record exactly once
-  Per-state transaction to external database is slow

9

treaming
What is Spark Streaming?
> Receive data streams from input sources, process
them in a cluster, push out to databases/
dashboards
> Scalable, fault-tolerant, second-scale latencies

Kafka
Flume HDFS

HDFS Databases
Kinesis treaming Dashboards
Twitter
How does Spark Streaming work?
>  Chop up data streams into batches of few secs
>  Spark treats each batch of data as RDDs and
processes them using RDD operations
>  Processed results are pushed out in batches

treamin
g
Receivers

data streams

batches as results as
RDDs RDDs
Spark Streaming Programming Model
> Discretized Stream (DStream)
-  Represents a stream of data
-  Implemented as a sequence of RDDs

> DStreams API very similar to RDD API
-  Functional APIs in Scala, Java
-  Create input DStreams from different sources
-  Apply parallel operations
Example – Get hashtags from Twitter
val  ssc  =  new  StreamingContext(sparkContext,  Seconds(1))  
val  tweets  =  TwitterUtils.createStream(ssc,  auth)  

Input  DStream  

Twi9er  Streaming  API   batch  @  t   batch  @  t+1   batch  @  t+2  

tweets  DStream  

stored  in  memory    

as  RDDs  
Example – Get hashtags from Twitter
val  tweets  =  TwitterUtils.createStream(ssc,  None)  
val  hashTags  =  tweets.flatMap(status  =>  getTags(status))  

transformed   transforma0on:  modify  data  in  one  
DStream   DStream  to  create  another  DStream    

batch  @  t   batch  @  t+1   batch  @  t+2  

tweets  DStream  

flatMap   flatMap   flatMap  

hashTags  Dstream  
…
new  RDDs  created  
[#cat,  #dog,  …  ]  
for  every  batch    
Example – Get hashtags from Twitter
val  tweets  =  TwitterUtils.createStream(ssc,  None)  
val  hashTags  =  tweets.flatMap(status  =>  getTags(status))  
hashTags.saveAsHadoopFiles("hdfs://...")  

output  opera0on:  to  push  data  to  external  storage  

batch  @  t   batch  @  t+1   batch  @  t+2  

tweets  DStream  
flatMap flatMap flatMap

hashTags  DStream  
save save save

every  batch  
saved  to  HDFS  
Example – Get hashtags from Twitter
val  tweets  =  TwitterUtils.createStream(ssc,  None)  
val  hashTags  =  tweets.flatMap(status  =>  getTags(status))  
hashTags.foreachRDD(hashTagRDD  =>  {  ...  })  

foreach:  do  whatever  you  want  with  the  processed  data  

batch  @  t   batch  @  t+1   batch  @  t+2  

tweets  DStream  
flatMap flatMap flatMap

hashTags  DStream  
foreach foreach foreach

Write  to  a  database,  update  analyOcs  

UI,  do  whatever  you  want  
Languages
Scala API  

val  tweets  =  TwitterUtils.createStream(ssc,  auth)  

val  hashTags  =  tweets.flatMap(status  =>  getTags(status))  

hashTags.saveAsHadoopFiles("hdfs://...”)

Java API
FuncOon  object  
JavaDStream<Status>  tweets  =  ssc.twitterStream()  
JavaDstream<String>  hashTags  =  tweets.flatMap(new  Function<...>  {    })  
hashTags.saveAsHadoopFiles("hdfs://...")  
 

Python API  
...soon  
Window-based Transformations
val  tweets  =  TwitterUtils.createStream(ssc,  auth)  
val  hashTags  =  tweets.flatMap(status  =>  getTags(status))  
val  tagCounts  =  hashTags.window(Minutes(1),  Seconds(5)).countByValue()  

sliding  window  
window  length   sliding  interval  
operaOon  

window  length  

DStream  of  data  

sliding  interval  
Arbitrary Stateful Computations
Specify function to generate new state based on
previous state and new data  
-  Example: Maintain per-user mood as state, and update it
with their tweets

   def  updateMood(newTweets,  lastMood)  =>  newMood  

 
   val  moods  =  tweetsByUser.updateStateByKey(updateMood  _)  

Arbitrary Combinations of Batch and
Streaming Computations

Inter-mix RDD and DStream operations!
-  Example: Join incoming tweets with a spam HDFS file to
filter out bad tweets"
 

 tweets.transform(tweetsRDD  =>  {  
 tweetsRDD.join(spamFile).filter(...)  
 })  

 
DStreams + RDDs = Power
> Combine live data streams with historical data
-  Generate historical data models with Spark, etc.
-  Use data models to process live data stream

> Combine streaming with MLlib, GraphX algos

-  Offline learning, online prediction
Spark     Spark   MLlib   GraphX  
-  Online learning and prediction SQL   Streaming   machine  
learning  
graph  
processing  

Apache  Spark  

> Interactively query streaming data using SQL

-  select * from table_from_streaming_data
Advantage of an Unified Stack
>  Explore data
$  ./spark-­‐shell  
scala>  val  file  =  sc.hadoopFile(“smallLogs”)  

interactively to
...  
scala>  val  filtered  =  file.filter(_.contains(“ERROR”))  

identify problems
...  
scala>  val  mapped  =  filtered.map(...)  
...  
object  ProcessProductionData  {  
     def  main(args:  Array[String])  {  
       val  sc  =  new  SparkContext(...)  
>  Use same code in        val  file  =  sc.hadoopFile(“productionLogs”)  
       val  filtered  =  file.filter(_.contains(“ERROR”))  
Spark for processing        val  mapped  =  filtered.map(...)  
       ...  
large logs    }  
}  object  ProcessLiveStream  {  
   def  main(args:  Array[String])  {  
       val  sc  =  new  StreamingContext(...)  
       val  stream  =  KafkaUtil.createStream(...)  
>  Use similar code in        val  filtered  =  stream.filter(_.contains(“ERROR”))  
       val  mapped  =  filtered.map(...)  
Spark Streaming for        ...  
   }  
realtime processing }  
Performance
Can process 60M records/sec (6 GB/sec) on
100 nodes at sub-second latency

7   3.5  
WordCount  
Cluster  Thhroughput  (GB/s)  

Cluster  Throughput  (GB/s)  

Grep  
6   3  
5   2.5  
4   2  
3   1.5  
2   1  
1  sec   1  sec  
1   0.5  
2  sec   2  sec  
0   0  
0   50   100   0   50   100  
#  Nodes  in  Cluster   #  Nodes  in  Cluster  
Fault-tolerance
> Batches of input data are tweets  
RDD   input  data  
replicated in memory for replicated  
fault-tolerance in  memory  

flatMap  
> Data lost due to worker
failure, can be recomputed
from replicated input data hashTags  
RDD  
lost  parOOons  
recomputed  on  
> All transformations are fault- other  workers  
tolerant, and exactly-once
transformations
Input Sources
•  Out of the box, we provide
-  Kafka, Flume, Kinesis, Raw TCP sockets, HDFS, etc.

•  Very easy to write a custom receiver

-  Define what to when receiver is started and stopped

•  Also, generate your own sequence of RDDs, etc.

and push them in as a “stream”

Output Sinks
•  HDFS, S3, etc (Hadoop API compatible filesystems)

•  Cassandra (using Spark-Cassandra connector)

•  Hbase (integrated support coming to Spark soon)

•  Directly push the data anywhere

Conclusion