Data Science with Hadoop: A Primer
ā¢
20 likesā¢6,797 views
Apache Hadoop is quickly becoming the technology of choice for organizations investing in big data, powering their next generation data architecture. With Hadoop serving as both a scalable data platform and computational engine, data science is re-emerging as a center-piece of enterprise innovation, with applied data solutions such as online product recommendation, automated fraud detection and customer sentiment analysis. In this talk Ofer will provide an overview of data science and how to take advantage of Hadoop for large scale data science projects: * What is data science? * How can techniques like classification, regression, clustering and outlier detection help your organization? * What questions do you ask and which problems do you go after? * How do you instrument and prepare your organization for applied data science with Hadoop? * Who do you hire to solve these problems? You will learn how to plan, design and implement a data science project with Hadoop
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to Data Science with Hadoop: A Primer
Similar to Data Science with Hadoop: A Primer (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Data Science with Hadoop: A Primer
- 1. Ā© Hortonworks Inc. 2013 Hortonworks Data Science with Hadoop ā A Primer Hadoop Summit, June 2013 Ofer Mendelevitch ofer@hortonworks.com @ofermend
- 2. Ā© Hortonworks Inc. 2013 Page 2 Who am I? currently <- c( role=ādirector of data sciencesā, company=āHortonworksā) ā¢ Previously: Nor1, Yahoo!, Risk Insight, Quiver, etcā¦ ā¢ Blog: www.achessdad.com
- 3. Ā© Hortonworks Inc. 2013 Page 3 What I will be talking about? ā¢What is Data Science? ā¢Hadoop and Data Science ā¢Use-cases: data science with Hadoop ā¢How to get started?
- 4. Ā© Hortonworks Inc. 2013 Page 4 What is Data Science? What is a data scientist? A person who does this Data Product: software product whose core functionality relies on applying statistical (or machine learning) methods to data. What is Data Science? The art of building data products
- 5. Ā© Hortonworks Inc. 2013 Page 5 Data science & big data
- 6. Ā© Hortonworks Inc. 2013 Page 6 With Hadoopā¦ Time and cost of building large scale data products is dramatically reduced
- 7. Ā© Hortonworks Inc. 2013 ApplianceCloudOS / VM An Apache Hadoop Platform HORTONWORKS DATA PLATFORM (HDP) PLATFORM SERVICES HADOOP CORE Enterprise Readiness: HA, DR, Snapshots, Security, ā¦ Distributed Storage & ProcessingHDFS MAP REDUCE DATA SERVICES Store, Process and Access Data HCATALOG HIVEPIG HBASE SQOOP FLUME OPERATIONAL SERVICES Manage & Operate at Scale OOZIE AMBARI
- 8. Ā© Hortonworks Inc. 2013 A typical Big Data Architecture Page 8 APPLICATIONSDATASYSTEMS TRADITIONAL REPOS RDBMS EDW MPP DATASOURCES MOBILE DATA OLTP, POS SYSTEMS OPERATIONAL TOOLS MANAGE & MONITOR Traditional Sources (RDBMS, OLTP, OLAP) New Sources (web logs, email, sensor data, social media) DEV & DATA TOOLS BUILD & TEST Business Analytics Custom Applications Packaged Applications HORTONWORKS DATA PLATFORM
- 9. Ā© Hortonworks Inc. 2013 Page 9 Keys to Hadoopās power ā¢ Computation co-located with data ā Data and computation system co-designed to work together ā¢ Affordable at scale ā Use ācommodityā hardware nodes ā Self-healing; failure handled by software ā Very good at batch processing of large datasets
- 10. Ā© Hortonworks Inc. 2013 Page 10 Hadoop improves productivity of data scientists ā¢All data in one place āAbility to store all the data in raw format āData silo convergence āData scientists will find innovative uses of combined data assets ā¢Data/compute capabilities available as shared asset āData scientists can quickly prototype a new idea without an up-front request for funding
- 11. Ā© Hortonworks Inc. 2013 Page 11 Data-driven innovation is accelerated since Hadoop is āschema on readā I need new data Finally, w e start collecting Let me seeā¦ is it any good? Start 6 months 9 months āSchema changeā project Letās just put it in a folder on HDFS Let me seeā¦ is it any good? 3 months My model is awesome!
- 12. Ā© Hortonworks Inc. 2013 Page 12 Hadoop is ideal for pre-processing of large raw datasets Strip away HTML/PDF/DOC/P PT Entity resolution Document vector generation Sampling, filtering Joins Raw Data Processed Data Term normalization
- 13. Ā© Hortonworks Inc. 2013 Page 13 In machine learning, very often: more data -> better outcomes Banko & Brill, 2001 ā¢More examples to learn from ā¢More possible feature types āWeāre looking for the most useful for our task
- 14. Ā© Hortonworks Inc. 2013 Page 14 Use-cases
- 15. Ā© Hortonworks Inc. 2013 Page 15 A (partial) map of data science ātasksā Discovery Clustering Detect natural groupings Outlier detection Detect anomalies Affinity Analysis Co-occurrence patterns Prediction Classification Predict a category Regression Predict a value Recommendation Predict a preference Big Data Science: High energy physics, Genomics, etc
- 16. Ā© Hortonworks Inc. 2013 Page 16 Use-case: product recommendation ā¢Inputs: āExplicit product ratings (when provided) āImplicit information: purchase transactions, page views, comments 5 2 4 ? ? ? ? 5 2 ? 1 2 ? ? 3 ? 2 3 1 5 Epic X-Men Hobbit Argo Pirates U101 U102 U103 U104 U105 ā¦ Ratings Page views Forum Comments
- 17. Ā© Hortonworks Inc. 2013 Page 17 Goal: predict a preference 5 2 4 ? ? ? ? 5 2 ? 1 2 ? ? 3 ? 2 3 1 5 Epic X-Men Hobbit Argo Pirates 5 2 4 1 3 4 1 5 2 3 1 2 4 1 3 3 2 3 1 5 U101 U102 U103 U104 U105 ā¦ U101 U102 U103 U104 U105 ā¦ Epic X-Men Hobbit Argo Pirates
- 18. Ā© Hortonworks Inc. 2013 Page 18 Using Hadoop for recommendation Pre-process SQL Online serving HDFS Map Reduce Transactions Page views Content Recommend Data sources Custom Logic With Hadoop, we can process very large preference datasets
- 19. Ā© Hortonworks Inc. 2013 Page 19 Use-case: failure prediction ā¢Inputs: āEquipment history: install date, model, past issues āEquipment sensor data āProduct catalog: product families, expected lifetime SKU Install date Service Person ID Zip code Avg temp TTF (days) 113454 5/1/2011 1345 94002 72 180 998323 5/3/2009 3234 88321 68 450 345375 8/2/2005 1112 53323 82 332 ā¦ ā¦ ā¦ ā¦ history Sensor data Product Catalog
- 20. Ā© Hortonworks Inc. 2013 Page 20 Building a prediction model SKU Install date Service Person ID Zip code Avg temp TTF (days) 113454 5/1/2011 1345 94002 72 180 998323 5/3/2009 3234 88321 68 450 345375 8/2/2005 1112 53323 82 332 ā¦ ā¦ ā¦ ā¦ Unseen data Model TTF Labeled Data SKU Install date Service Person ID Zip code Avg temp 332456 3/3/2013 1345 94005 71 442343 6/6/2013 1112 77485 67
- 21. Ā© Hortonworks Inc. 2013 Page 21 Using Hadoop for failure prediction ā¢ HDFS: central repository for all data ā Service records (word, pdf, etc) ā Equipment purchase transaction data ā Product catalog: SKUs, model numbers, etc ā¢ Pre-process ā Convert service records to item features: remove PDF formatting, detect entities in records ā Normalize data using service records, product catalog ā Create feature matrix; ready for modeling algorithm
- 22. Ā© Hortonworks Inc. 2013 Page 22 Use-case: SaaS application security ā¢Inputs: āClick-stream: user interaction with application User ID User since Logins/m onth Avg DL KB/day ā¦ 123456 1/3/2004 6 30 998323 5/3/2009 1 5 345375 8/2/2005 22 120 ā¦ ā¦ ā¦ ā¦ User data Clicks
- 23. Ā© Hortonworks Inc. 2013 Page 23 Detecting anomalous behavior records ā¢ User access profile modeled as vector of features ā¢ Detect anomalies in application access patterns ā Rules based ā Machine learning based (determine āoutlier factorā: 0ā¦1)
- 24. Ā© Hortonworks Inc. 2013 Page 24 Using Hadoop for anomaly detection ā¢ HDFS: central repository for all raw data ā Raw user-access logs ā User information (organization, demographics) ā¢ Pre-process ā Build access-profile (behavioral) for each user ā¢ Detect anomalies ā In Hadoop ā Using existing tools: R, SAS, rules engine, etc
- 25. Ā© Hortonworks Inc. 2013 Page 25 How do I get started?
- 26. Ā© Hortonworks Inc. 2013 Page 26 1. Pick a good use-case that delivers immediate business value 2. Implement a proof-of-value (POV) 3. Build a team (hire/train) Getting started with Data science on Hadoop
- 27. Ā© Hortonworks Inc. 2013 Page 27 ā¢ Put together a Hadoop cluster ā¢ Define the POV business use-case ā¢ Pull raw data you need into the cluster ā¢ Build it ā¢ Show the business value of your data assets Contact us. We can help! Implement a proof-of-value
- 28. Ā© Hortonworks Inc. 2013 Page 28 Build a team: The data scientist skillset continuum Software engineer Research Scientist Data Engineer Data Scientist Applied Scientist Role Data Engineer Applied Scientist Function Builds production-grade data products Finds signal/meaning in the data Applies statistical/ML models and tunes the algorithm Good atā¦. Data and Systems architecture Hadoop, PIG/HIVE, MapReduce, mahout Java, Python, Perl, SQL, C++, etc NoSQL (Hbase, Cassandra, Mongo) Statistics, Machine learning Text processing, NLP R, Matlab, SAS, SQL Sciptring, prototyping Visualization / telling the story
- 29. Ā© Hortonworks Inc. 2013 Page 29 Thank you! Any Questions? Ofer Mendelevitch Director, Data Sciences @ Hortonworks ofer@hortonworks.com @ofermend Weāre hiring! Data Science training: www.hortonworks.com/training
Editor's Notes
- Data science is not new. But now we need to do it with much larger datasets.
- As the volume of data has exploded, we increasingly see organizations acknowledge that not all data belongs in a traditional database. The drivers are both cost (as volumes grow, database licensing costs can become prohibitive) and technology (databases are not optimized for very large datasets).Instead, we increasingly see Hadoop ā and HDP in particular ā being introduced as a complement to the traditional approaches. It is not replacing the database but rather is a complement: and as such, must integrate easily with existing tools and approaches. This means it must interoperate with:Existing applications ā such as Tableau, SAS, Business Objects, etc,Existing databases and data warehouses for loading data to / from the data warehouseDevelopment tools used for building custom applicationsOperational tools for managing and monitoring