[Neutron] Slow network speed between VM and external

Recently I encountered a situation of the VM's network performance that is pretty low. For instance, if upload a image to Glance via my VM to run glance cli command, the actual transmit speed is around the following data:

RX: 130 Kbps

TX: 150 Kbps

My OpenStack's network service is with GRE segmentation on Neutron. Why? After google the web, I found some people had the same issues as I had:

https://ask.openstack.org/en/question/25306/slow-network-speed-between-vm-and-external/

They talked about the approach on the VM to solve the issue and this works

> ethtool -K eth0 gro off

> ethtool -K eth0 tso off

But, for my case I think it is not solve from the root cause so that I take time to do the research for TCP segmentation offload and GRE. Finally I find the root cause:
MTU need to adjust when using GRE network service.
http://docs.openstack.org/juno/install-guide/install/yum/content/neutron-network-node.html
Quote:
"Tunneling protocols such as GRE include additional packet headers that increase overhead and decrease space available for the payload or user data. Without knowledge of the virtual network infrastructure, instances attempt to send packets using the default Ethernet maximum transmission unit (MTU) of 1500 bytes.Internet protocol (IP) networks contain the path MTU discovery (PMTUD) mechanism to detect end-to-end MTU and adjust packet size accordingly. However, some operating systems and networks block or otherwise lack support for PMTUD causing performance degradation or connectivity failure.
Ideally, you can prevent these problems by enabling jumbo frames on the physical network that contains your tenant virtual networks. Jumbo frames support MTUs up to approximately 9000 bytes which negates the impact of GRE overhead on virtual networks. However, many network devices lack support for jumbo frames and OpenStack administrators often lack control over network infrastructure. Given the latter complications, you can also prevent MTU problems by reducing the instance MTU to account for GRE overhead. Determining the proper MTU value often takes experimentation, but 1454 bytes works in most environments. You can configure the DHCP server that assigns IP addresses to your instances to also adjust the MTU."

After doing the following setting on Controller node, the transmit speed is up to:

RX: 31323 Kbps

TX: 31464 Kbps

1. Edit the /etc/neutron/dhcp_agent.ini file and complete the following action:
   1. In the [DEFAULT] section, enable the dnsmasq configuration file:
      

      1 2 3

      [DEFAULT] ... dnsmasq_config_file = /etc/neutron/dnsmasq-neutron.conf

2. Create and edit the /etc/neutron/dnsmasq-neutron.conf file and complete the following action:
   1. Enable the DHCP MTU option (26) and configure it to 1454 bytes:
      

      1	dhcp-option-force=26,1454

3. Kill any existing dnsmasq processes:

   # pkill dnsmasq

So, the network performance is improved 200 times.
Wow...



Reference:
TCP segmentation offload
https://forum.ivorde.com/linux-tso-tcp-segmentation-offload-what-it-means-and-how-to-enable-disable-it-t19721.html
TCP in Linux Kernel
http://vger.kernel.org/~davem/tcp_output.html
Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environmenthttps://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140

[OpenStack] The resource list for studying OpenStack Neutron

The following list is about the resource documents for those who want to study OpenStack Neutron. It may reduce the time you spend on searching. Here you go:

Introduction to OpenStack Quantum

• To warm up for jumping into the network world of OpenStack

OpenStack Admin Guide Chapter 7. Networking

• It is very important to take a look at this offical document first

Neurton Wiki

Architecture of Quantum Grizzly Release

• This contains the Quantum architecture in details

A Cisco Plugin Framework for Quantum L2 Network Overlays Spanning Multiple Physical Switches

• Cicso's Plugins are a little bit complicated because There are several versions those are different in the configuration and prerequisite.

Cisco Nexus Plug-in for OpenStack Neutron 

• Give a data sheet to list the functionality and feature it supports

What's new Neutron?

• Give a overall look of Neutron
• Another doc: http://www.slideshare.net/kamesh001/whats-new-in-neutron-for-open-stack-havana
• What components are in Neutron: http://www.slideshare.net/emaganap/open-stack-overview-meetups-oct-2013

Deep Dive into Neutron (Important)

• It is very technical to explain the the detail of code and message.

Open vSwitch in Neutron (Important)

• we can see how Open vSwitch is implemented in Neutron 

OpenStack Overview

• It introduces VM booting workflow with Nova and Networking
• Neurton deployment components

OPENSTACK QUANTUM (NEUTRON) PLUG-IN: THERE CAN ONLY BE ONE

• To explain why ML2 will come out

Modular Layer 2 in OpenStack Neutron

• New Feature: ToR Switch Control
• The video: http://www.youtube.com/watch?v=whmcQ-vHams

ML2

• Modular Layer 2 (ML2) The Modular Layer 2 (ML2) plugin is a new OpenSource plugin to Neutron. This plugin is a framework allowing OpenStack Networking to simultaneously utilize the variety of layer 2 networking technologies found in complex real-world data centers. It currently works with the existing Open vSwitch, Linux Bridge, and L2 agents. The ML2 plugin supports local, flat, VLAN, GRE and VXLAN network types via a type drivers and different mechanism drivers.

OVSDB User Guide

• OpenDaylight has OVSDB project that is related with ML2 plugin

Cisco OpenStack Overview

• Introduce Cisco Neutron Plugin
• Cisco Virtual Switch: Nexus 1000v
• Cisco UCSM

OpenStack Networking API v2.0 Reference

OpenStack RDO Deployment 

1. http://blog.csdn.net/cloudtech/article/details/19936249
2. http://blog.csdn.net/cloudtech/article/details/19936425
3. http://blog.csdn.net/cloudtech/article/details/19936487

OverStack over VXLAN

• https://wiki.openstack.org/wiki/Arista-neutron-ml2-driver
• Arista Related Information about Mechanism Driver
• https://blueprints.launchpad.net/neutron/+spec/arista-ml2-mechanism-driver
• http://www.bradreese.com/blog/4-1-2013-2.pdf
• https://docs.google.com/document/d/1efFprzY69h-vaikRE8hoGQuLzOzVNtyVZZLa2GHbXLI/edit

Developer Guide

How to write a Neutron Plugin

Neutron/LBaaS/PluginDrivers

http://www.slideshare.net/MiguelLavalle/network-virtualization-with-open-stack-quantum