PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
Spark Streaming, Machine Learning, Graph Processing and Approximations with Kinesis
1. Spark and Friends:
Spark Streaming,
Machine Learning, Graph Processing,
Lambda Architecture, Approximations
Chris Fregly
East Bay Java User Group
Oct 2014
Kinesis
Streaming
2. Who am I?
Former Netflix’er
(netflix.github.io)
Spark Contributor
(github.com/apache/spark)
Founder
(fluxcapacitor.com)
Author
(effectivespark.com,
sparkinaction.com)
10. Spark Overview
• Based on 2007 Microsoft Dryad paper
• Written in Scala
• Supports Java, Python, SQL, and R
• Data fits in memory when possible, but not
required
• Improved efficiency over MapReduce
– 100x in-memory, 2-10x on-disk
• Compatible with Hadoop
– File formats, SerDes, and UDFs
14. Benefits of Unified Libraries
• Advancements in higher-level libraries are
pushed down into core and vice-versa
• Examples
– Spark Streaming
• GC and memory management improvements
– Spark GraphX
• IndexedRDD for random, hashed-based access within
a partition versus scanning entire partition
– Spark Core
• Sort-based Shuffle
16. RDD Overview
• Core Spark abstraction
• Represents partitions
across the cluster nodes
• Enables parallel processing
on data sets
• Partitions can be in-memory
or on-disk
• Immutable, recomputable,
fault tolerant
• Contains transformation history (“lineage”) for
whole data set
20. Demo!
• Load user data
• Load gender data
• Join user data
with gender data
• Analyze lineage
21. Join Optimizations
• When joining large dataset with small dataset (reference
data)
• Broadcast small dataset to each node/partition of large
dataset (one broadcast per node)
23. Spark API Overview
• Richer, more expressive than MapReduce
• Native support for Java, Scala, Python,
SQL, and R (mostly)
• Unified API across all libraries
• Operations
– Transformations (lazy evaluation)
– Actions (execute transformations)
27. Job Scheduling
• Job
– Contains many stages
• Contains many tasks
• FIFO
– Long-running jobs may starve resources for other
jobs
• Fair
– Round-robin to prevent resource starvation
• Fair Scheduler Pools
– High-priority pool for important jobs
– Separate user pools
– FIFO within the pool
– Modeled after Hadoop Fair Scheduler
28. Spark Execution Model Overview
• Parallel, distributed
• DAG-based
• Lazy evaluation
• Allows optimizations
– Reduce disk I/O
– Reduce shuffle I/O
– Single pass through dataset
– Parallel execution
– Task pipelining
• Data locality and rack awareness
• Worker node fault tolerance using RDD
lineage graphs per partition
35. Master High Availability
• Multiple Master Nodes
• ZooKeeper maintains current Master
• Existing applications and workers will be
notified of new Master election
• New applications and workers need to
explicitly specify current Master
• Alternatives (Not recommended)
– Local filesystem
– NFS Mount
39. Goal: Increase Matches (1/2)
• Top Influencers
– PageRank
– Most desirable people
• People You May Know
– Shortest Path
– Facebook-enabled
• Recommendations
– Alternating Least
Squares (ALS)
48. Spark Streaming API Overview
• Rich, expressive API similar to core
• Operations
– Transformations (lazy)
– Actions (execute transformations)
• Window and State Operations
• Requires checkpointing to snip long-running
DStream lineage
• Register DStream as a Spark SQL table
for querying?! Wow.
62. Characteristics of Sources
• Buffered
– Flume, Kafka, Kinesis
– Allows replay and back pressure
• Batched
– Flume, Kafka, Kinesis
– Improves throughput at expense of duplication on
failure
• Checkpointed
– Kafka, Kinesis
– Allows replay from specific checkpoint
63. Message Delivery Guarantees
• Exactly once [1]
– No loss
– No redeliver
– Perfect delivery
– Incurs higher latency for transactional semantics
– Spark default per batch using DStream lineage
– Degrades to less guarantees depending on source
• At least once [1..n]
– No loss
– Possible redeliver
• At most once [0,1]
– Possible loss
– No redeliver
– *Best configuration if some data loss is acceptable
• Ordered
– Per partition: Kafka, Kinesis
– Global across all partitions: Hard to scale
64. Types of Checkpoints
Spark
1. Spark checkpointing of StreamingContext
DStreams and metadata
2. Lineage of state and window DStream
operations
Kinesis
3. Kinesis Client Library (KCL) checkpoints
current position within shard
– Checkpoint info is stored in DynamoDB per
Kinesis application keyed by shard
65. Fault Tolerance
• Points of Failure
– Receiver
– Driver
– Worker/Processor
• Possible Solutions
– Use HDFS File Source for durability
– Data Replication
– Secondary/Backup Nodes
– Checkpoints
• Stream, Window, and State info
66. Streaming Receiver Failure
• Use a backup receiver
• Use multiple receivers pulling from multiple
shards
– Use checkpoint-enabled, sharded streaming
source (ie. Kafka and Kinesis)
• Data is replicated to 2 nodes immediately
upon ingestion
– Will spill to disk if doesn’t fit in memory
• Possible loss of most-recent batch
• Possible at-least once delivery of batch
• Use buffered sources for replay
– Kafka and Kinesis
67. Streaming Driver Failure
• Use a backup Driver
– Use DStream metadata checkpoint info to
recover
• Single point of failure – interrupts stream
processing
• Streaming Driver is a long-running Spark
application
– Schedules long-running stream receivers
• State and Window RDD checkpoints to
HDFS to help avoid data loss (mostly)
68. Stream Worker/Processor Failure
• No problem!
• DStream RDD partitions will be recalculated from lineage
• Causes blip in processing during node failover
72. Tuning
• Batch interval
– High: reduce overhead of submitting new tasks for each batch
– Low: keeps latencies low
– Sweet spot: DStream job time (scheduling + processing) is
steady and less than batch interval
• Checkpoint interval
– High: reduce load on checkpoint overhead
– Low: reduce amount of data loss on failure
– Recommendation: 5-10x sliding window interval
• Use DStream.repartition() to increase parallelism of processing
DStream jobs across cluster
• Use spark.streaming.unpersist=true to let the Streaming Framework
figure out when to unpersist
• Use CMS GC for consistent processing times
79. Approximation Overview
• Required for scaling
• Speed up analysis of large datasets
• Reduce size of working dataset
• Data is messy
• Collection of data is messy
• Exact isn’t always necessary
• “Approximate is the new Exact”
81. Approximations In Action
Figure: Memory Savings with Approximation Techniques
(http://highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/)
82. Spark Statistics Library
• Correlations
– Dependence between 2 random variables
– Pearson, Spearman
• Hypothesis Testing
– Measure of statistical significance
– Chi-squared test
• Stratified Sampling
– Sample separately from different sub-populations
– Bernoulli and Poisson sampling
– With and without replacement
• Random data generator
– Uniform, standard normal, and Poisson distribution
83. Summary
• Spark, Spark Streaming Overview
• Use Cases
• API and Libraries
• Machine Learning
• Graph Processing
• Execution Model
• Fault Tolerance
• Cluster Deployment
• Monitoring
• Scaling and Tuning
• Lambda Architecture
• Probabilistic Data Structures
• Approximations
http://effectivespark.com http://sparkinaction.com
Thanks!!
Chris Fregly
@cfregly