Hadoop, Pig, and Twitter (NoSQL East 2009)
•
300 likes•143,205 views
A talk on the use of Hadoop and Pig inside Twitter, focusing on the flexibility and simplicity of Pig, and the benefits of that for solving real-world big data problems.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Hadoop, Pig, and Twitter (NoSQL East 2009)
Similar to Hadoop, Pig, and Twitter (NoSQL East 2009) (20)
Recently uploaded
Recently uploaded (20)
Hadoop, Pig, and Twitter (NoSQL East 2009)
- 1. Hadoop, Pig, and Twitter Kevin Weil -- @kevinweil Analytics Lead, Twitter TM
- 2. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
- 3. My Background ‣ Studied Mathematics and Physics at Harvard, Physics at Stanford ‣ Tropos Networks (city-wide wireless): mesh routing algorithms, GBs of data ‣ Cooliris (web media): Hadoop and Pig for analytics, TBs of data ‣ Twitter: Hadoop, Pig, machine learning, visualization, social graph analysis, ??? of data
- 4. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
- 5. Data is Getting Big ‣ NYSE: 1 TB/day ‣ Facebook: 20+ TB compressed/day ‣ CERN/LHC: 40 TB/day (15 PB/year!) ‣ And growth is accelerating ‣ Need multiple machines, horizontal scalability
- 6. Hadoop ‣ Distributed file system (hard to store a PB) ‣ Fault-tolerant, handles replication, node failure, etc ‣ MapReduce-based parallel computation (even harder to process a PB) ‣ Generic key-value based computation interface allows for wide applicability ‣ Open source, top-level Apache project ‣ Scalable: Y! has a 4000-node cluster ‣ Powerful: sorted a TB of random integers in 62 seconds
- 7. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 8. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 9. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 10. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 11. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 12. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 13. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
- 14. But... ‣ Analysis typically done in Java ‣ Single-input, two-stage data flow is rigid ‣ Projections, filters: custom code ‣ Joins: lengthy, error-prone ‣ n-stage jobs: Hard to manage ‣ Prototyping/exploration requires ‣ analytics in Eclipse? compilation ur doin it wrong...
- 15. Enter Pig ‣ High level language ‣ Transformations on sets of records ‣ Process data one step at a time ‣ Easier than SQL?
- 16. Why Pig? ‣ Because I bet you can read the following script.
- 17. A Real Pig Script ‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.
- 18. No, seriously.
- 19. Pig Democratizes Large-scale Data Analysis ‣ The Pig version is: ‣ 5% of the code ‣ 5% of the time ‣ Within 50% of the execution time. ‣ Innovation increasingly driven from large-scale data analysis ‣ Need fast iteration to understand the right questions ‣ More minds contributing = more value from your data
- 20. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
- 21. MySQL, MySQL, MySQL ‣ We all start there. ‣ But MySQL is not built for analysis. ‣ select count(*) from users? Maybe. ‣ select count(*) from tweets? Uh... ‣ Imagine joining them. ‣ And grouping. ‣ Then sorting.
- 22. The Hadoop Ecosystem at Twitter ‣ Cloudera’s free distribution, running Hadoop 0.20.1 ‣ Heavily modified Facebook Scribe for log collection -> HDFS* ‣ Heavily modified LZO code for fast, splittable data compression** ‣ Data stored either as LZO-compressed flat files (logs, etc) or serialized, LZO-compressed protocol buffers (structured data). ‣ Custom InputFormats, Pig LoadFuncs for the above* ‣ Some Java-based MapReduce, some Hadoop Streaming ‣ Most analysis, and most interesting analyses, done in Pig. ‣ * Open sourced, or on the way. Please come talk afterwards if you’re interested. ‣ ** http://www.github.com/kevinweil/hadoop-lzo
- 23. Data? ‣ Semi-structured: apache logs, search logs, RoR logs, mysql query logs, rate limiter logs, per-application logs ‣ Structured: tweets, users, block notifications, phones, favorites, saved searches, retweets, authentications, sms usage, third party clients, followings ‣ Entangled: the social graph
- 24. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
- 25. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣
- 26. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣
- 27. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣
- 28. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣
- 29. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣ How many unique queries, how many unique users? ‣
- 30. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣ How many unique queries, how many unique users? ‣ What is their geographic distribution?
- 31. Counting Big Data ‣ Where are users querying from? The API, the front page, their profile page, etc? ‣
- 32. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users?
- 33. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients?
- 34. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses
- 35. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time?
- 36. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked?
- 37. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often?
- 38. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often? ‣ Search corrections, search suggestions
- 39. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often? ‣ Search corrections, search suggestions ‣ A/B testing
- 40. Correlating Big Data ‣ What is the correlation between users with registered phones and users that tweet?
- 41. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets?
- 42. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow?
- 43. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers?
- 44. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following?
- 45. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following? ‣ What graph structures lead to successful networks?
- 46. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following? ‣ What graph structures lead to successful networks? ‣ User reputation
- 47. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis
- 48. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted?
- 49. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree?
- 50. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection
- 51. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning
- 52. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning ‣ Language detection
- 53. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning ‣ Language detection ‣ ... the list goes on.
- 54. Research on Big Data ‣ How well can we detect bots and other non-human tweeters?
- 55. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
- 56. Why Hadoop? ‣ Data is growing rapidly; need horizontally scalable computation ‣ Fault tolerant; gracefully handles machine failure ‣ ... but writing MapReduce jobs in Java is harder than it should be.
- 57. Why Pig? ‣ Pig makes Hadoop accessible ‣ Pig chains together complex job flows ‣ User-defined functions are first class citizens ‣ Vibrant OS community, dedicated team at Y! improving it daily ‣ At Twitter, Pig helps us understand our business faster.
- 58. Questions? Follow me at twitter.com/kevinweil ‣ If this sounded interesting to you -- that’s because it is. And we’re hiring. TM