SlideShare a Scribd company logo

分散Key-valueストアkumofsの思想と設計

•
•
Sadayuki Furuhashi
Sadayuki Furuhashi

丸レクセミナー2010 分散Key-valueストア「kumofs」について紹介いたします。近年では、スケーラビリティを重視した分散データストアが数多く登場してきました。その中でkumofsは、特に可用性の高さや動的な運用のしやすさを重視しており、また、高い単体性能を誇ります。このセッションではkumofsの特性についてご紹介すると共に、稼働中のサーバの追加やサーバ故障時の復旧方法などのデモを交えながら、kumofsの運用について具体的に紹介いたします。

Technology
1 of 72
Download to read offline
kumofs
mpio MessagePack
100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
Server Server Manager Gateway
Tokyo Cabinet Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
Server Server Server Server Server Server
Server Manager Gateway
Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
Server Server Server Manager Server Server Manager Gateway Application
Server Server Server Manager Server Server Manager Gateway Application
Server Server Server Manager Server Server Manager Server Gateway Application
Gateway Application
Server Server Server localhost:11211 Gateway RPC / Application
Server Server Server Manager
Manager Manager Server Server svr001 svr002
Manager Manager Server Server Server svr001 svr002 svr003
Manager Manager Server Server Server svr001 svr002 svr003 Server svr005
Manager Manager Server svr001 svr002 svr003 Server Server Server svr005 svr006 svr007
110,000 req/sec 114,000 req/sec 114,000 85,500 Linux 2.6.27.10 x86_64 AMD Athlon 64 X2 5000+ 57,000 Linux 2.6.27.19 x86_64 AMD Athlon 64 X2 4800+ 28,500 key 32 value 1KB 8 32 key 960,000 key 0 memcached kumofs
(requests/sec) 3,200,000 kumofs 2,400,000 Linux 2.6.27.21 i686 Intel Pentium 4 3.20GHz Linux 2.6.27.21 x86_64 Intel QuadCore Xeon X3350 1,600,000 50 10 60 key 8 value 32 32 800,000 256 key 51,200,000 key 0 10 20 30 40 50 60 ( )
Demo

Recommended

NoSQLを知る
NoSQLを知るNoSQLを知る
NoSQLを知るSadayuki Furuhashi
 
Ethereum blockchain
Ethereum blockchainEthereum blockchain
Ethereum blockchainMad Devs
 
Docker-OVS
Docker-OVSDocker-OVS
Docker-OVSsnrism
 
Misc
MiscMisc
MiscShamz Deez
 
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiIp sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiHuy Eav
 
愛のSuperCollider
愛のSuperCollider愛のSuperCollider
愛のSuperColliderMasayuki Akamatsu
 
An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)Naoto MATSUMOTO
 
Openv switchの使い方とか
Openv switchの使い方とかOpenv switchの使い方とか
Openv switchの使い方とかkotto_hihihi
 

Recommended

NoSQLを知る
NoSQLを知るNoSQLを知る
NoSQLを知るSadayuki Furuhashi
 
Ethereum blockchain
Ethereum blockchainEthereum blockchain
Ethereum blockchainMad Devs
 
Docker-OVS
Docker-OVSDocker-OVS
Docker-OVSsnrism
 
Misc
MiscMisc
MiscShamz Deez
 
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiIp sec vpn with dynamic routing mikrotik and cisco - mikro-tik wiki
Ip sec vpn with dynamic routing mikrotik and cisco - mikro-tik wikiHuy Eav
 
愛のSuperCollider
愛のSuperCollider愛のSuperCollider
愛のSuperColliderMasayuki Akamatsu
 
An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)An Easy way to build a server cluster without top of rack switches (MEMO)
An Easy way to build a server cluster without top of rack switches (MEMO)Naoto MATSUMOTO
 
Openv switchの使い方とか
Openv switchの使い方とかOpenv switchの使い方とか
Openv switchの使い方とかkotto_hihihi
 
Containers on Windows
Containers on WindowsContainers on Windows
Containers on WindowsPratik Khasnabis
 
How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineRitesh Rushiya
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]Naoto MATSUMOTO
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSRyo ONODERA
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingNaoto MATSUMOTO
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5Vasanth Reddy
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
 
Netmiko library
Netmiko libraryNetmiko library
Netmiko libraryManjunath annure
 
Docker Networking
Docker NetworkingDocker Networking
Docker NetworkingWeaveworks
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化Yoko Nakano
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloValerio Balbi
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpnAshwajit Maske
 
ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースSadayuki Furuhashi
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728jieun kim
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyLeif Hedstrom
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOIJohnson Liu
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Yongyoon Shin
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04Hiroshi SHIBATA
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSazuredayit
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014Hiroshi SHIBATA
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogicRakuten Group, Inc.
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)markmcclain
 

More Related Content

What's hot

How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineRitesh Rushiya
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]Naoto MATSUMOTO
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSRyo ONODERA
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingNaoto MATSUMOTO
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5Vasanth Reddy
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
 
Docker Networking
Docker NetworkingDocker Networking
Docker NetworkingWeaveworks
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化Yoko Nakano
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloValerio Balbi
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpnAshwajit Maske
 

What's hot (12)

Containers on Windows
Containers on WindowsContainers on Windows
Containers on Windows
 
How to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command lineHow to configure v mware v switch from esx-esxi command line
How to configure v mware v switch from esx-esxi command line
 
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]IPv4 over IPv6 Tunneling with IPSec [DRAFT]
IPv4 over IPv6 Tunneling with IPSec [DRAFT]
 
FreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPSFreeBSD under DigitalOcean VPS
FreeBSD under DigitalOcean VPS
 
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 TunnelingUNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
UNDOCUMENTED Vyatta vRouter: IPv4 over IPv6 Tunneling
 
Esx versions diff 3.5
Esx versions diff 3.5Esx versions diff 3.5
Esx versions diff 3.5
 
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
 
Netmiko library
Netmiko libraryNetmiko library
Netmiko library
 
Docker Networking
Docker NetworkingDocker Networking
Docker Networking
 
初めての仮想化
初めての仮想化初めての仮想化
初めての仮想化
 
FreeLix: Semplicità & Controllo
FreeLix: Semplicità & ControlloFreeLix: Semplicità & Controllo
FreeLix: Semplicità & Controllo
 
3 manual installation of open vpn
3 manual installation of open vpn3 manual installation of open vpn
3 manual installation of open vpn
 

Similar to 分散Key-valueストアkumofsの思想と設計

ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースSadayuki Furuhashi
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728jieun kim
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyLeif Hedstrom
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOIJohnson Liu
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Yongyoon Shin
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04Hiroshi SHIBATA
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSazuredayit
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014Hiroshi SHIBATA
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogicRakuten Group, Inc.
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)markmcclain
 
Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio Soares
 
New idc architecture
New idc architectureNew idc architecture
New idc architectureMason Mei
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVMJohn Lee
 
Betting On Data Grids
Betting On Data GridsBetting On Data Grids
Betting On Data Gridsgojkoadzic
 
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewCloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewChip Childers
 
Run Run Trema Test
Run Run Trema TestRun Run Trema Test
Run Run Trema TestHiroshi Ota
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Joe Arnold
 
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamSharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamOfficience
 
Learn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceLearn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceOpenCity Community
 

Similar to 分散Key-valueストアkumofsの思想と設計 (20)

ずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケースずばり動く！kumofs と ずばり動かないケース
ずばり動く！kumofs と ずばり動かないケース
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
 
Usenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a ProxyUsenix LISA 2012 - Choosing a Proxy
Usenix LISA 2012 - Choosing a Proxy
 
Olive Introduction for TOI
Olive Introduction for TOIOlive Introduction for TOI
Olive Introduction for TOI
 
Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1Harmonia open iris_basic_v0.1
Harmonia open iris_basic_v0.1
 
20140419 oedo rubykaigi04
20140419 oedo rubykaigi0420140419 oedo rubykaigi04
20140419 oedo rubykaigi04
 
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKSAzure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
Azure Day Rome Reloaded 2019 - Deconstructing Kubernetes using AKS
 
20140425 ruby conftaiwan2014
20140425 ruby conftaiwan201420140425 ruby conftaiwan2014
20140425 ruby conftaiwan2014
 
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
[Rakuten TechConf2014] [C-5] Ichiba Architecture on ExaLogic
 
Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)Intro to OpenStack Astara (Spring '16)
Intro to OpenStack Astara (Spring '16)
 
Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11Livio slides-libflexsc-usenix-atc11
Livio slides-libflexsc-usenix-atc11
 
New idc architecture
New idc architectureNew idc architecture
New idc architecture
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVM
 
Dfhghg
DfhghgDfhghg
Dfhghg
 
Betting On Data Grids
Betting On Data GridsBetting On Data Grids
Betting On Data Grids
 
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 PreviewCloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
CloudStack DC Meetup - Apache CloudStack Overview and 4.1/4.2 Preview
 
Run Run Trema Test
Run Run Trema TestRun Run Trema Test
Run Run Trema Test
 
Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012Swift Install Workshop - OpenStack Conference Spring 2012
Swift Install Workshop - OpenStack Conference Spring 2012
 
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday VietnamSharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
SharePoint 2010’s Virtual Reality - SharePoint Saturday Vietnam
 
Learn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceLearn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practice
 

More from Sadayuki Furuhashi

Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Sadayuki Furuhashi
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesSadayuki Furuhashi
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Sadayuki Furuhashi
 
Fluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupFluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupSadayuki Furuhashi
 
DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？Sadayuki Furuhashi
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container EraSadayuki Furuhashi
 
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11Sadayuki Furuhashi
 
Fighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkFighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkSadayuki Furuhashi
 
Embulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダSadayuki Furuhashi
 
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsPlugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsSadayuki Furuhashi
 
Embulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderEmbulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderSadayuki Furuhashi
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Sadayuki Furuhashi
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreSadayuki Furuhashi
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi
 

More from Sadayuki Furuhashi (20)

Scripting Embulk Plugins
Scripting Embulk PluginsScripting Embulk Plugins
Scripting Embulk Plugins
 
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019
 
Making KVS 10x Scalable
Making KVS 10x ScalableMaking KVS 10x Scalable
Making KVS 10x Scalable
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics Pipelines
 
Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理Digdagによる大規模データ処理の自動化とエラー処理
Digdagによる大規模データ処理の自動化とエラー処理
 
Fluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes MeetupFluentd at Bay Area Kubernetes Meetup
Fluentd at Bay Area Kubernetes Meetup
 
DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？DigdagはなぜYAMLなのか？
DigdagはなぜYAMLなのか？
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
 
Fighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkFighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with Embulk
 
Embulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダ
 
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsPlugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGems
 
Embuk internals
Embuk internalsEmbuk internals
Embuk internals
 
Embulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderEmbulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loader
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
 
Prestogres internals
Prestogres internalsPrestogres internals
Prestogres internals
 
Presto+MySQLで分散SQL
Presto+MySQLで分散SQLPresto+MySQLで分散SQL
Presto+MySQLで分散SQL
 
Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014Presto - Hadoop Conference Japan 2014
Presto - Hadoop Conference Japan 2014
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect More
 
Prestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for PrestoPrestogres, ODBC & JDBC connectivity for Presto
Prestogres, ODBC & JDBC connectivity for Presto
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfdanishmna97
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfOverkill Security
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?Paolo Missier
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data SciencePaolo Missier
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdfMuhammad Subhan
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfAnubhavMangla3
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 

Recently uploaded (20)

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdf
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 

分散Key-valueストアkumofsの思想と設計

  • 2. mpio MessagePack
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. 100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
  • 18. 100% 10% 1% 0% 1 2 4 8 16 32 64 128 256 512
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37. Server Server Manager Gateway
  • 38. Tokyo Cabinet Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
  • 39. Server Server Server Server Server Server
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 56. Server Server Server Manager Server Server Manager Server Gateway Gateway Gateway Application Application Application
  • 57. Server Server Server Manager Server Server Manager Gateway Application
  • 58. Server Server Server Manager Server Server Manager Gateway Application
  • 59. Server Server Server Manager Server Server Manager Server Gateway Application
  • 61. Server Server Server localhost:11211 Gateway RPC / Application
  • 62.
  • 63. Server Server Server Manager
  • 64. Manager Manager Server Server svr001 svr002
  • 65. Manager Manager Server Server Server svr001 svr002 svr003
  • 66. Manager Manager Server Server Server svr001 svr002 svr003 Server svr005
  • 67. Manager Manager Server svr001 svr002 svr003 Server Server Server svr005 svr006 svr007
  • 68. 110,000 req/sec 114,000 req/sec 114,000 85,500 Linux 2.6.27.10 x86_64 AMD Athlon 64 X2 5000+ 57,000 Linux 2.6.27.19 x86_64 AMD Athlon 64 X2 4800+ 28,500 key 32 value 1KB 8 32 key 960,000 key 0 memcached kumofs
  • 69. (requests/sec) 3,200,000 kumofs 2,400,000 Linux 2.6.27.21 i686 Intel Pentium 4 3.20GHz Linux 2.6.27.21 x86_64 Intel QuadCore Xeon X3350 1,600,000 50 10 60 key 8 value 32 32 800,000 256 key 51,200,000 key 0 10 20 30 40 50 60 ( )
  • 70.
  • 71.
  • 72. Demo