Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013
•
145 likes•86,537 views
"This is a technical architect's case study of how Loggly has employed the latest social-media-scale technologies as the backbone ingestion processing for our multi-tenant, geo-distributed, and real-time log management system. This presentation describes design details of how we built a second-generation system fully leveraging AWS services including Amazon Route 53 DNS with heartbeat and latency-based routing, multi-region VPCs, Elastic Load Balancing, Amazon Relational Database Service, and a number of pro-active and re-active approaches to scaling computational and indexing capacity. The talk includes lessons learned in our first generation release, validated by thousands of customers; speed bumps and the mistakes we made along the way; various data models and architectures previously considered; and success at scale: speeds, feeds, and an unmeltable log processing engine."
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013
Similar to Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013
- 1. Unmeltable Infrastructure at Scale: Using Apache Kafka, Twitter Storm, and ElasticSearch on AWS Jim Nisbet Philip O’Toole CTO and VP of Engineering, Loggly Lead Developer, Infrastructure, Loggly November 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
- 2. What Loggly Does • Log Management as a service – Near real-time indexing of events • Distributed architecture, built on AWS • Initial production services in 2010 – Loggly Generation 2 released in Sept 2013 • Thousands of customers
- 3. Agenda for this Presentation • • • • • A bit about logging Lessons learned from our first generation How we leverage AWS services Our use of Kafka, Storm, ElasticSearch What worked well for us and what did not
- 4. Log Management • Everyone starts with … – A bunch of log files (syslog, application specific) – On a bunch of machines • Management consists of doing the simple stuff – Rotate files, compress and delete – Information is there but awkward to find specific events – Weird log retention policies evolve over time
- 5. “…how can I make this someone else’s problem!” “…hmmm, our logs are getting a bit bloated” Log Volume Self-Inflicted Pain “…let’s spend time managing our log capacity”
- 6. Best Practices in Log Management • Use existing logging infrastructure – Real time syslog forwarding is built in – Application log file watching • Store logs externally – Accessible when there is a system failure • Log messages in machine parsable format – JSON encoding when logging structured information – Key-value pairs
- 7. From the Trenches… • Managing Applications vs. Managing Logs – Do not make this is an either/or proposition! If you get a disk space alert, first login… % sudo rm –rf /var/log/apache2/* Admit it, we’ve all seen this kind of thing!
- 8. You Have Logs... 2013-10-25T18:35:43.387+0000: 441.482: [GC [PSYoungGen: 2430541K->268617K(2484544K)] 7687523K->5660738K(8076992K), 0.3266870 secs] [Times: user=1.05 sys=0.17, real=0.33 secs] 2013-10-25T18:35:43.714+0000: 441.809: [Full GC [PSYoungGen: 268617K->0K(2484544K)] [ParOldGen: 5392121K->354965K(5592448K)] 5660738K->354965K(8076992K) [PSPermGen: 44444K->44395K(83968K)], 0.9225290 secs] [Times: user=2.22 sys=0.26, real=0.92 secs] • In this case, JVM garbage collection logs enabled with… -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
- 9. Yes, you need to search these logs
- 10. But you also need to to spot trends
- 11. Loggly Offers Logging as a Service
- 12. Loggly First Generation • Logging as a service – Near real-time searchable logs • Thousands of customers – Transmission rates from 10 events/sec to 100k events/sec – When customers systems are busy they send more logs – Log traffic has distinct bursts; bursts can last for several hours • Amazon EC2 deployment – We used EC2 Instance storage • SOLR Cloud – Full power of Lucene search – Tens of thousands of shards (with special ‘sleep shard’ logic) • ZeroMQ for message queue
- 13. First Generation Lessons Learned • Event ingestion too tightly coupled to indexing – Manual re-indexing for temporary SOLR issues • Multiple Indexing strategies needed – 4 orders of magnitude difference between our high volume users and our low volume users (10 eps vs. 100,000+ eps) – Too much system overhead for low volume users – Difficult to support changing indexing strategies for a customer
- 14. Big Data Infrastructure Solutions We are not alone… • Our challenges – – – – – Massive incoming event stream Fundamentally multi-tenant Scalable framework for analysis Near real-time indexing Time series index management Scalability Real Time Analytics Multi tenant SaaS
- 15. Apache Kafka • Overview – – – – An Apache project initially developed at LinkedIn Distributed publish-subscribe messaging system Specifically designed for real time activity streams Does not follow JMS Standards nor uses JMS APIs • Key Features – – – – Persistent messaging High throughput, low overhead Uses ZooKeeper for forming a cluster of nodes Supports both queue and topic semantics
- 17. Storm Framework • Storm (open sourced by Twitter) – Open sourced September 2011 – Now an Apache Software Foundation project • Currently Incubator Status • Framework is for stream processing – – – – Distributed Fault tolerant Computation Fail-fast components
- 18. Storm Logical View Example Topology Bolt Spout Bolt Bolt Bolt Spouts emit source stream Bolts perform stream processing Storm terminology • Streams, Spouts, Bolts, Tasks, Workers, Stream Groups and Topologies
- 20. ElasticSearch • Open source – Commercial support available from ElasticSearch.com – Growing open-source community • • • • • Distributed search engine Fully exposes Lucene search functionality Built for clustering from the ground-up High availability Multi-tenancy
- 21. ElasticSearch In Action • Add/delete nodes dynamically • Add indices with REST API • Indices and Nodes have attributes – Indices automatically moved to best Nodes • Indices can be sharded • Supports bulk insertion of events • Plugins for monitoring cluster
- 23. Generation 2 – The Challenge • Always accept log data – Never make a customer’s incident worse • Never drop log data – A single log message could be critical • True Elasticity
- 24. Perfect Match For Real Time Log Events • Apache Kafka – Extremely high-performance pub-sub persistent queue • Consumer tracks their location in queue – A good fit for our use cases • Multiple Kafka brokers – Good match for AWS • Multiple brokers per region • Availability Zone separation
- 25. Real Time Event Processing • Twitter Storm – Scalable real-time computation system • Storm used as a “pull” system – Provisioned for average load, not peak load – Input from Kafka queue • Worker nodes can be scaled dynamically • Elasticity is key – Another good match for AWS • Able to scale workers up and down dynamically
- 27. Kafka Queue
- 28. Loggly Collector Performance • C++ multi-threaded • Boost ASIO framework • Each Collector can handle 250k+ events per second – Per m2.2xlarge instance 1 x EC2 m2.2xlarge Collector instance (300 byte average event size).
- 31. Event Pipeline in Summary • Storm provides Complex Event Processing – Where we run much of our secret-sauce • Kafka contains both raw and processed Events • Snapshot the last day of Kafka events to S3
- 32. Elastic Search Clusters From Kafka Multi-Tiered Elastic Cluster
- 33. Loggly and Index Management • Indices are time-series data – Separated by customer – Represent slices of time • Higher volume index will have shorter time slice • Multi-tier architecture for efficient indexing – Multiple indexing tiers mapped to different AWS instance types • Efficient use of AWS resources
- 36. Kafka enables Staging Architecture • Kafka Broker doesn’t care if there are multiple consumers • Staging system runs pre-production code • Pub-sub allows us to randomly index a fraction of our production load • A highly-effective pre-production system
- 38. AWS Deployment Instances – Collection c1.xlarge m2.2xlarge • • • • • Compute-optimized High-traffic ingestion points Disk not important Memory-optimized Disk buffer caching 4K Provisioned IOPs EBS • • • Ensures consistent IO No noisy-neighbors Persistent storage
- 39. AWS Deployment Instances – Processing ZooKeeper c1.xlarge • • • Compute-optimized CPU-intensive processing Network IO m1.xlarge • • • General-purpose Configuration Management
- 40. AWS Deployment Instances – Indexing cc2.8xlarge • 4K Provisioned IOPs EBS m2.4xlarge • 4K Provisioned IOPs EBS
- 41. A Few False Starts
- 42. Elastic Load Balancing in front of Collector Had Limitations • Initial testing used Elastic Load Balancing for incoming events: • Elastic Load Balancing doesn’t allow forwarding port 514 (syslog) • Elastic Load Balancing doesn’t support forwarding UDP • Event traffic can burst and hit Elastic Load Balancing performance limits
- 43. Amazon Route 53 DNS Round Robin a Win • DNS Round Robin is pretty basic load balancing – Not a bump in the wire • Take advantage of AWS failover health checks – When a collector goes out of service, it will be out of the DNS rotation • Round Robin across multiple regions, AZs – Latency based resolution optimizes inbound traffic
- 44. Our First Plan for Log Events • Cassandra – Highly scalable key-value store – Impressive write performance a good match for us – Apache project plus commercial support with DataStax • Use Cassandra for both our Event Queue and Persistent Store – Our strategy was to get the raw event in to Cassandra – …then perform workflow processing on events
- 45. Design meets Reality • Cassandra not designed to be a message queue • Hard to track Events received out-of-order • Multi-tenancy requires handling data bursts – Collectors still needed to be able to buffer to disk – Added complexity and became a point of failure
- 46. Big Wins • Leveraging AWS services – – – – Multi-region, multi-AZ Provisioned IOPS for availability and scale Amazon Route 53 DNS support with latency resolution Easy to increase and decrease Storm resources • Leveraging Open Source infrastructure – Apache Kafka – Twitter Storm – ElasticSearch • Pre-production “Staging” system
- 47. The Means to an End
- 48. Feedback • Questions? Jim Nisbet (niz@loggly.com) CTO and VP of Engineering, Loggly Philip O’Toole (philip@loggly.com) Lead Developer, Infrastructure, Loggly Follow us @loggly!
- 49. Please give us your feedback on this presentation ARC303 As a thank you, we will select prize winners daily for completed surveys!