Sunday, August 31, 2014

[Lagopus] Install Lagopus software switch on Ubuntu 12.04

 I attended NTT ( Ryu/Lagopus ) seminar in Aug. at NCTU, Taiwan and noticed that Lagopus(SDN/OpenFlow Software Switch) is amazing.
Its L2 switch performance with 10GbE X 2 (RFC2889 test ) for most of packet size are near to 10Gbps and the test platform is 
Intel xeon E5-2660 (8 cores, 16 threads), Intel X520-DA2 DDR3-1600 64GB. 
For more information please see the attachment pictures that I took from the seminar.

The following is the features:
  • Best OpenFlow 1.3 compliant software-based switch
    • Multi tables, Group tables support
    • MPLS, PBB, QinQ, support
  • ONF standard specification support
    • OpenFlow Switch Specification 1.3.3
    • OF-CONFIG 1.1
  • Multiple data-plane configuration
    • High performance software data-plane on Intel x86 bare-metal server
      • Intel DPDK, Raw socket
    • Bare metal switch
  • Various management/configuration interfaces
    • OF-CONFIG, OVSDB, CLI
    • SNMP, Ethernet-OAM functionality

For installing Lagopus switch, you can refer to the following URL. It can give us a common installation guide for Lagopus switch.
https://github.com/lagopus/lagopus/blob/master/QUICKSTART.md


About my lagopus environment:sudo vi /usr/local/etc/lagopus/lagopus.conf
 interface {  
   ethernet {  
     eth0;  
     eth1;  
     eth2;  
   }  
 }  
 bridge-domains {  
   br0 {  
     port {  
       eth0;  
       eth1;  
       eth2;  
     }  
     controller {  
       127.0.0.1;  
     }  
   }  
 }  


But, I put 2 shell scripts for quickly installing and setting up the DPDK. I use DPDK-1.6.0 so that all the scripts are based on this version.

compile_dpdk.sh
 #!/bin/sh  
 export RTE_SDK=/home/myname/git/DPDK-1.6.0  
 export RTE_TARGET="x86_64-default-linuxapp-gcc"  
 make config T=${RTE_TARGET}  
 make install T=${RTE_TARGET}  

install_dpdk.sh
 #!/bin/sh  
 export RTE_SDK=/home/myname/git/DPDK-1.6.0
 export RTE_TARGET="x86_64-default-linuxapp-gcc"  
 DPDK_NIC_PCIS="0000:00:08.0 0000:00:09.0 0000:00:0a.0"  
 HUGEPAGE_NOPAGES="1024"  
 set_numa_pages()  
 {  
     for d in /sys/devices/system/node/node? ; do  
         sudo sh -c "echo ${HUGEPAGE_NOPAGES} > $d/hugepages/hugepages-2048kB/nr_hugepages"  
     done  
 }  
 set_no_numa_pages()  
 {  
     sudo sh -c "echo ${HUGEPAGE_NOPAGES} > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages"  
 }  
 # install module  
 sudo modprobe uio  
 sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/igb_uio.ko  
 sudo insmod ${RTE_SDK}/${RTE_TARGET}/kmod/rte_kni.ko  
 # unbind e1000 NICs from igb and bind igb_uio for DPDK  
 sudo ${RTE_SDK}/tools/pci_unbind.py --bind=igb_uio ${DPDK_NIC_PCIS}  
 sudo ${RTE_SDK}/tools/pci_unbind.py --status  
 # mount fugepagefs  
 echo "Set hugepagesize=${HUGEPAGE_NOPAGES} of 2MB page"  
 NCPUS=$(find /sys/devices/system/node/node? -maxdepth 0 -type d | wc -l)  
 if [ ${NCPUS} -gt 1 ] ; then  
     set_numa_pages  
 else  
     set_no_numa_pages  
 fi  
 echo "Creating /mnt/huge and mounting as hugetlbfs"  
 sudo mkdir -p /mnt/huge  
 grep -s '/mnt/huge' /proc/mounts > /dev/null  
 if [ $? -ne 0 ] ; then  
     sudo mount -t hugetlbfs nodev /mnt/huge  
 fi  
 unset RTE_SDK  
 unset RTE_TARGET  

Here is one thing needs to be notice. 
The variable DPDK_NIC_PCIS is my Linux eth1, eth2, and eth3's bus info as follows:
DPDK_NIC_PCIS="0000:00:08.0 0000:00:09.0 0000:00:0a.0"
You have to change them by running ethtool to see your eth bus info.

So, we need to use the comand "ethtool" to find out the NIC's bus-info as follows:
# ethtool -i eth4
driver: igb
version: 5.0.5-k
firmware-version: 3.11, 0x8000046e
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no


After executing the 2 shell scripts, then we can start the lagopus switch by this:
sudo lagopus -d -- -c3 -n1 -- -p3


4 comments:

Abhik said...

I executed the 2 scripts (modified the env variables and NICs accordingly), but when I try to run lagopus it is throwing error that "Cannot init TX queue 0 for port 0 (-22)".

I updated the lagopus.conf in /usr/local/etc/lagopus/lagopus.conf as follows:

interface {
ethernet {
eth0;
eth1;
eth2;
eth3;
}
}
bridge-domains {
br0 {
port {
eth0;
eth1;
eth2;
eth3;
}
controller {
192.168.2.51;
}
}
}

sudo ${RTE_SDK}/tools/pci_unbind.py --status gives the following output.

Network devices using IGB_UIO driver
====================================
0000:04:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3
0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3
0000:13:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3
0000:1b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3 unused=igb_uio *Active*

Other network devices
=====================



When I try to run lagopus using "sudo lagopus -d -- -c3 -n1 -- -p15", it is giving the following error:

.........
Initializing NIC port 0 ...
Initializing NIC port 0 RX queue 0 ...
Initializing NIC port 0 TX queue 0 ...
PANIC in app_init_nics():
Cannot init TX queue 0 for port 0 (-22)
.........

Can you help me to find out what am I doing wrong?

TeYen (Danny) said...

The value of the variable DPDK_NIC_PCIS should be modified to your interface bus info. For my example, I reserve the Linux eth0 and put Linux eth1, eth2 and eth3's bus info as 0000:00:08.0, 0000:00:09.0, and 0000:00:0a.0
You can use ethtool to check your interface's bus info.

Unknown said...
This comment has been removed by the author.
Unknown said...

Could you plz tell me how to use Ryu controller with lagopus via Openflow Protocol!!! I have read ryubook and I am trying to implement SDN in DPDK using RYU controller and Lagopus datapath in Ubuntu 14.04/12. Is it necessary to start/run RYU using mininet. I am new in DPDK so if you know where am i wrong then plz correct me. I am following this paper http://lagopus.github.io/handson/handson.pdf