High order bits from cassandra & hadoop

•Download as PPTX, PDF•

0 likes•932 views

srisatish ambati

Technology

Thank You! svccg in first page of search results for “cloud” on google!

points Usecases Why cassandra? Usecase: Hadoop, Brisk FUD:Consistency Why facebook is not using Cassandra? Anti-patterns Community, Code, Tools Q&A

Users. Netflix. Key by Customer, read-heavy Key by Customer:Movie, write-heavy

TimeSeries: (several customers) periodic readings: dev0, dev1…deviceID:metric:timestamp ->value Metrics typically way larger dataset than users.

Replication: Multi-datacenter Multi-region ec2 Multi-availability zones

reads local dc1 dc2 Replication: Multi-datacenter Multi-region ec2, aws Multi-availability zones

4.21.2011, Amazon Web Services outage: “Movie marathons on Netflix awaiting AWS to come back up.” #ec2disabled

4.21.2011, Amazon Web Services outage: Netflix was running on AWS.

Writes Sequential, append-only. ~1-5ms On cloud: ephemeral disks rock!

Reads Local Key & row caches, (also, jna-based 0xffheap) indexes, materialized

Reads Local Key & row caches, (also, jna-based 0xffheap) indexes, materialized ssds: improved read performance!

Distribution between nodes Gossip Anti-entropy Failure-detector L i g h t w e i g h t

Clients: cql, thrift pycassa, phpcassa hector, pelops (scala, ruby, clojure)

Usecase #3: hadoop Hdfs cassandra hive Logs stats analytics

jobtracker, tasktracker hdfs: namenode, datanode

cloudera amazon: elastic map reduce hortonworks mapR brisk

Tools & Analytics Hive, Pig, R Karmasphere Datameer … dozens of stealth startups!

Use column families (tables) inode sblock

near-real time hadoop Low latency: cassandra_dc nodes Batch Analytics: brisk_dc nodes

Consistency: R + W > N ORACLE, 2-node: R=1, W=2, N=2,(T=2) DNS * N is replication factor. Not to be confused with T=total #of nodes

Tune-able, flexibility. For High Consistency: read:quorum, write:quorum For High Availability: high W, low R.

Inbox Search: 600+cores.120+TB (2008) Went from 100-500m users. Average NoSQL deployment size: ~6-12 nodes.

Usecase #5: search Apache Solr + Cassandra = Solandra Other inbox/file Searches: xobni, c3 github.com/tjake/solandra

“Eventual consistency is harder to program.” mostly immutable data. complex systems at scale.

Miscellaneous, Myth: data-loss, partial rows. writes are durable.

Anti-Patterns Transactions Joins Read before write

Anti-Patterns for cloud ebs jvm, virtualized single region

Tools AMIs, OpsCenter, DataStax AppDynamics Netflix just builds AMIs for deployment!

B e a u t i f u l C 0 d e = new code(); //less is more ~90k.java.concurrent.@annotate. bloomfilters, merkletrees. non-blocking, staged-event-driven. bigtable, dynamo.

Current & Future Focus: Distributed Counters, CQL. Simple client. operational smoothening. compaction.

Community Robust. Rapid. # Professional support from DataStax. Filesysteminnovatin from Acunu engineers: independent,startups, large companies, Rackspace, Twitter, Netflix.. Come join the efforts!

Usecase #4: first NoSQL, then scale! simpledb Cassandra mongodb Cassandra

Copyright: plantoys … more than one way to do it!

Summary - high scale peer-to-peer datastore best friend for multi-region, multi-zone availability. Hadoop – HDFS engulfing the DataWorld

What's hot

The Automation FactoryNathan Milford

Lessons from Cassandra & Spark (Matthias Niehoff & Stephan Kepser, codecentri...DataStax

Spark application on ec2 clusterChao-Hsuan Shen

Cassandra CLuster Management by Japan Cassandra CommunityHiromitsu Komatsu

SSTable Reader Cassandra Day Denver 2014Ben Vanberg

Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...Data Con LA

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)Spark Summit

Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax

Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...DataStax

Instaclustr Webinar 50,000 Transactions Per Second with Apache Spark on Apach...Instaclustr

Learn Cassandra at edureka!Edureka!

Introduction to CassandraGokhan Atil

Instaclustr webinar 2017 feb 08 japanHiromitsu Komatsu

Up and running with pysparkKrishna Sangeeth KS

Scylla db deck, july 2017Dor Laor

Introduction to Cassandra: Replication and ConsistencyBenjamin Black

Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016DataStax

Time to Science, Time to Results. Accelerating Scientific research in the CloudAmazon Web Services

From DataFrames to Tungsten: A Peek into Spark's Future-(Reynold Xin, Databri...Spark Summit

Introduction to cassandra 2014Patrick McFadin

What's hot (20)

The Automation Factory

Lessons from Cassandra & Spark (Matthias Niehoff & Stephan Kepser, codecentri...

Spark application on ec2 cluster

Cassandra CLuster Management by Japan Cassandra Community

SSTable Reader Cassandra Day Denver 2014

Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...

Optimizing Your Cluster with Coordinator Nodes (Eric Lubow, SimpleReach) | Ca...

Instaclustr Webinar 50,000 Transactions Per Second with Apache Spark on Apach...

Learn Cassandra at edureka!

Introduction to Cassandra

Instaclustr webinar 2017 feb 08 japan

Up and running with pyspark

Scylla db deck, july 2017

Introduction to Cassandra: Replication and Consistency

Lambda Architecture with Cassandra (Vaibhav Puranik, GumGum) | C* Summit 2016

Time to Science, Time to Results. Accelerating Scientific research in the Cloud

From DataFrames to Tungsten: A Peek into Spark's Future-(Reynold Xin, Databri...

Introduction to cassandra 2014

Similar to High order bits from cassandra & hadoop

Netflix and Open SourceAdrian Cockcroft

MySQL Cluster Scaling to a Billion QueriesBernd Ocklin

Spring one2gx2010 spring-nonrelational_dataRoger Xia

Apache Cassandra Lunch #72: Databricks and CassandraAnant Corporation

Cassandra & Python - Springfield MO User GroupAdam Hutson

Navigating NoSQL in cloudy skiesshnkr_rmchndrn

On Rails with Apache CassandraStu Hood

Apache cassandra and spark. you got the the lighter, let's start the firePatrick McFadin

C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...DataStax Academy

Breakthrough OLAP performance with Cassandra and SparkEvan Chan

Big data vahidamiri-tabriz-13960226-datastack.irdatastack

Spinnaker VLDB 2011sandeep_tata

Developing with CassandraSperasoft

Introduction to parallel iterative deep learning on hadoop’s next generation...Anh Le

No SqlMichael Marth

A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson

Minnebar 2013 - Scaling with CassandraJeff Bollinger

NoSql DatabaseSuresh Parmar

CassandraUpaang Saxena

Similar to High order bits from cassandra & hadoop (20)

Netflix and Open Source

MySQL Cluster Scaling to a Billion Queries

Spring one2gx2010 spring-nonrelational_data

Apache Cassandra Lunch #72: Databricks and Cassandra

Cassandra & Python - Springfield MO User Group

Navigating NoSQL in cloudy skies

On Rails with Apache Cassandra

Apache cassandra and spark. you got the the lighter, let's start the fire

C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...

Breakthrough OLAP performance with Cassandra and Spark

Big data vahidamiri-tabriz-13960226-datastack.ir

Spinnaker VLDB 2011

Developing with Cassandra

Introduction to parallel iterative deep learning on hadoop’s next generation...

No Sql

A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala

Minnebar 2013 - Scaling with Cassandra

NoSql Database

Cassandra

Recently uploaded

Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3

Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

Data governance with Unity Catalog PresentationKnoldus Inc.

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes

Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq

Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani

So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda

Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda

TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey

Scale your database traffic with Read & Write split using MySQL RouterMydbops

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

UiPath Community: Communication Mining from Zero to HeroUiPathCommunity

Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen

The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech

Recently uploaded (20)

Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...

The State of Passkeys with FIDO Alliance.pptx

[Webinar] SpiraTest - Setting New Standards in Quality Assurance

Glenn Lazarus- Why Your Observability Strategy Needs Security Observability

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx

Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...

Time Series Foundation Models - current state and future directions

Data governance with Unity Catalog Presentation

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

Genislab builds better products and faster go-to-market with Lean project man...

Potential of AI (Generative AI) in Business: Learnings and Insights

So einfach geht modernes Roaming fuer Notes und Nomad.pdf

Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger

TeamStation AI System Report LATAM IT Salaries 2024

Scale your database traffic with Read & Write split using MySQL Router

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx

UiPath Community: Communication Mining from Zero to Hero

Testing tools and AI - ideas what to try with some tool examples

The Ultimate Guide to Choosing WordPress Pros and Cons

High order bits from cassandra & hadoop

1. High-order bits from Cassandra & Hadoop srisatishambati @srisatish

2. Thank You! svccg in first page of search results for “cloud” on google!

3. NoSQL- Know your queries.

4. points Usecases Why cassandra? Usecase: Hadoop, Brisk FUD:Consistency Why facebook is not using Cassandra? Anti-patterns Community, Code, Tools Q&A

5. Users. Netflix. Key by Customer, read-heavy Key by Customer:Movie, write-heavy

6. TimeSeries: (several customers) periodic readings: dev0, dev1…deviceID:metric:timestamp ->value Metrics typically way larger dataset than users.

7. Why Cassandra?

8. Operational simplicity peer-to-peer

9. Operational simplicity peer-to-peer

10. Replication: Multi-datacenter Multi-region ec2 Multi-availability zones

11. reads local dc1 dc2 Replication: Multi-datacenter Multi-region ec2, aws Multi-availability zones

12. 4.21.2011, Amazon Web Services outage: “Movie marathons on Netflix awaiting AWS to come back up.” #ec2disabled

13. 4.21.2011, Amazon Web Services outage: Netflix was running on AWS.

14. fast durable writes. fast reads.

15. Writes Sequential, append-only. ~1-5ms

16. Writes Sequential, append-only. ~1-5ms On cloud: ephemeral disks rock!

17. Reads Local Key & row caches, (also, jna-based 0xffheap) indexes, materialized

18. Reads Local Key & row caches, (also, jna-based 0xffheap) indexes, materialized ssds: improved read performance!

19. Distribution between nodes Gossip Anti-entropy Failure-detector L i g h t w e i g h t

20. Clients: cql, thrift pycassa, phpcassa hector, pelops (scala, ruby, clojure)

21. Usecase #3: hadoop Hdfs cassandra hive Logs stats analytics

22. Brisk Truly peer-to-peer hadoop.

23. mv computation not data

24.

25. Parallel Execution View

26.

27. jobtracker, tasktracker hdfs: namenode, datanode

28. cloudera amazon: elastic map reduce hortonworks mapR brisk

29. Tools & Analytics Hive, Pig, R Karmasphere Datameer … dozens of stealth startups!

30. Namenode decomposition, explained.

31.

32.

33. Use column families (tables) inode sblock

34. near-real time hadoop Low latency: cassandra_dc nodes Batch Analytics: brisk_dc nodes

35. FUD, acronym: fear, uncertainty, doubt.

36. Consistency: R + W > N ORACLE, 2-node: R=1, W=2, N=2,(T=2) DNS * N is replication factor. Not to be confused with T=total #of nodes

37. Tune-able, flexibility. For High Consistency: read:quorum, write:quorum For High Availability: high W, low R.

38.

39. Inbox Search: 600+cores.120+TB (2008) Went from 100-500m users. Average NoSQL deployment size: ~6-12 nodes.

40. Usecase #5: search Apache Solr + Cassandra = Solandra Other inbox/file Searches: xobni, c3 github.com/tjake/solandra

41. “Eventual consistency is harder to program.” mostly immutable data. complex systems at scale.

42. Miscellaneous, Myth: data-loss, partial rows. writes are durable.

43. Anti-Patterns Transactions Joins Read before write

44. Anti-Patterns for cloud ebs jvm, virtualized single region

45. Three good reasons for Cassandra...

46. Tools AMIs, OpsCenter, DataStax AppDynamics Netflix just builds AMIs for deployment!

47. B e a u t i f u l C 0 d e = new code(); //less is more ~90k.java.concurrent.@annotate. bloomfilters, merkletrees. non-blocking, staged-event-driven. bigtable, dynamo.

48. Current & Future Focus: Distributed Counters, CQL. Simple client. operational smoothening. compaction.

49. Community Robust. Rapid. # Professional support from DataStax. Filesysteminnovatin from Acunu engineers: independent,startups, large companies, Rackspace, Twitter, Netflix.. Come join the efforts!

50.

51. Usecase #4: first NoSQL, then scale! simpledb Cassandra mongodb Cassandra

52.

53.

54.

55. Copyright: xkcd

56. Copyright: plantoys … more than one way to do it!

57. Summary - high scale peer-to-peer datastore best friend for multi-region, multi-zone availability. Hadoop – HDFS engulfing the DataWorld

58. Q&A @srisatish

59. NoSQL- Know your queries.

High order bits from cassandra & hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to High order bits from cassandra & hadoop

Similar to High order bits from cassandra & hadoop (20)

More from srisatish ambati

More from srisatish ambati (11)

Recently uploaded

Recently uploaded (20)

High order bits from cassandra & hadoop