A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2
•
21 likes•5,017 views
A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2 29 Nov, 2012 SAKURA Internet Research Center. Senior Researcher / Naoto MATSUMOTO
Recommended
Recommended
More Related Content
More from Naoto MATSUMOTO
More from Naoto MATSUMOTO (20)
Recently uploaded
Recently uploaded (20)
A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2
- 1. 29 Nov, 2012 SAKURA Internet Research Center Senior Researcher / Naoto MATSUMOTO
- 2. VXLAN TESTING Overview 1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al. http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html +config VXLAN + tristate "Virtual eXtensible Local Area Network (VXLAN)" + depends on EXPERIMENTAL + ---help--- + This allows one to create vxlan virtual interfaces that provide + Layer 2 Networks over Layer 3 Networks. VXLAN is often used + to tunnel virtual network infrastructure in virtualized environments. + For more information see: + http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02 2) Infiniband Adapter is the fastest of all for datacenter networking. Dual-port Infiniband FDR 56Gbit/s Host Bus Adapter VXLAN + IP over Infiniband Network is AWESOME!.
- 3. Buildup to the New Kernel & etc 1) Install OFED for Infiniband Networking # wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz # tar xzvf ./OFED-1.5.4.1.tgz # ./OFED-1.5.4.1/install.pl /*Install OFED software */ # sync; sync; sync; reboot; 2) Linux Kernel Update # cd /usr/local/src # wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2 # tar xvfj linux-3.7-rc7.tar.bz2 # cd linux-3.7-rc7 # cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config # make oldconfig # vi .config CONFIG_VXLAN=m /* Add New Line */ # make; make modules_install ; make install # sync; sync; sync; reboot 3) iproute2 Update # yum install db4-devel.x86_64 # git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git # cd iproute2; ./configure ; make ; make install Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 4. VXLAN Configuration # uname -a Linux VXLAN-A 3.7.0-rc7 #1 SMP .... 56Gbit/s IPoIB # ip addr add 10.1.1.1/24 dev ib1 IP over Infiniband Network # ip route add 224.0.0.0/4 dev ib1 # ip link set up dev ib1 # ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 VXLAN Network # ip addr add 192.168.99.1/24 dev vxlan99 # ip link set up dev vxlan99 # iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */ # uname -a Linux VXLAN-B 3.7.0-rc7 #1 SMP .... # ip addr add 10.1.1.2/24 dev ib1 # ip route add 224.0.0.0/4 dev ib1 # ip link set up dev ib1 # ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 # ip addr add 192.168.99.2/24 dev vxlan99 # ip link set up dev vxlan99 # iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */ Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 5. VXLAN performance check VXLAN-A# ip link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc 56Gbit/s IPoIB link/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff IP over Infiniband Network VXLAN-A# iperf -c 192.168.99.2 -d : VXLAN Network [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/sec VXLAN-B# ip link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff VXLAN-B# iperf -s & vnstat -l -i ib1 Monitoring ib1... (press CTRL-C to stop) rx: 18.49 Gbit/s 53431 p/s tx: 22.82 Gbit/s 58258 p/s Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 6. Traffic Analysis using Wireshark Outer Header Inner Header Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 7. Debug TIPS for tshark 1) VXLAN Traffic Monitoring # tshark -pni ib1 Capturing on ib1 0.000000 10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472 0.001072 10.1.1.2 -> 10.1.1.1 UDP Source port: 39994 Destination port: 8472 2) IGMPv3 Join/Releave Monitoring # tshark -i ib1 -VV Frame 1 (56 bytes on wire, 56 bytes captured) Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22) Internet Group Management Protocol [IGMP Version: 3] Type: Membership Report (0x22) Header checksum: 0xea9a [correct] Num Group Records: 1 Group Record : 239.0.0.99 Change To Exclude Mode Record Type: Change To Exclude Mode (4) Aux Data Len: 0 Num Src: 0 Multicast Address: 239.0.0.99 (239.0.0.99) Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 8. Debug TIPS for iproute2 1) VXLAN Link status # ip -d link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300 2) VXLAN IP Address information # ip -d addr show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff inet 192.168.99.1/24 scope global vxlan99 inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link valid_lft forever preferred_lft forever 3) VXLAN Network Table Lookup # netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.1.1.0 0.0.0.0 255.255.255.0 U 00 0 ib1 192.168.99.0 0.0.0.0 255.255.255.0 U 00 0 vxlan99 224.0.0.0 0.0.0.0 240.0.0.0 U 00 0 ib1 Source: SAKURA Internet Research Center. 11/2012: Project THORN
- 9. Thank you for your interest.