[Open Source] A list of useful Python Open Source Projects

Tornado is a Python web framework and asynchronous networking library
http://www.tornadoweb.org/en/stable/

a powerful interactive packet manipulation program
http://www.secdev.org/projects/scapy/

[NSX] Network Visualization Gets Physical

In VMware NSX, how to let network visualization gets physical?
The answer is "Layer 2 Gateway Services". The physical switches have to implement the OVSDB and related APIs that communicate with NSX Controller so that they can provide the mapping of the segment in virtual network and VLAN in physical network. For more information in details, you can check out the following URLs.

http://networkheresy.com/2013/08/15/network-virtualization-gets-physical/


https://www.youtube.com/watch?v=QDOlggwyrVA&feature=c4-overview-vl&list=PLdYldEmmLm2lz5Bd0bzGCDKSULL52ytJT

Ivan's NSX Architecture Introduction, it is very good.
http://demo.ipspace.net/get/NSXArch

[OpenVNet] The introduction of OpenVNet

Why I pay attention to OpenVNet is because it uses Trema-Edge as OpenFlow 1.3 Controller to communicate with Open vSwitch 1.10. It gives us a great example to use Trema-Edge in Virtual Network Environment and learn more about some use cases. Please check out the following lists:

What is OpenVNet?
http://www.slideshare.net/akirayokokawa/openvnet

Offical Web Site:
http://openvnet.com/

Github Source:
https://github.com/axsh/openvnet

[Quagga] How to compile and install Quagga on Ubuntu 12.04

The following steps are the compilation and installation for Quagga on Ubuntu 12.04. Hopefully it is helpful for you.

• Install all dependency(package) 
• sudo apt-get build-dep quagga
• Give the path for all configurations files and libraries used by quagga for starting. I use this one /opt/quagga .... and this is done by this comand: 
• ./configure --enable-vtysh --prefix=/opt/quagga --localstatedir=/opt/quagga sysconfdir=/opt/quagga 
• Create new folder: 
• sudo mkdir /opt/quagga
• Add new user : 
• sudo adduser quagga 
• Give the priviledge for the user quagga over the folder: /opt/quagga and for the folders and subfolders you give the privilegde for all rights : reading, changing and executing
• sudo chown quagga:quagga /opt/quagga
• sudo chmod 777 /opt/quagga 
• Then you will be able to comiple Quagga: 
• make
• sudo make install 

Quagga configuration
--------------------
quagga version          : 0.99.22
host operating system   : linux-gnu
source code location    : .
compiler                : gcc
compiler flags          : -Os -fno-omit-frame-pointer -g -std=gnu99 -Wall -Wsign-compare -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wmissing-prototypes -Wmissing-declarations -Wchar-subscripts -Wcast-qual
make                    : make
includes                : 
linker flags            :  -lcrypt   -lrt -lcap  -lm
state file directory    : /opt/quagga
config file directory   : /opt/quagga
example directory       : /opt/quagga
user to run as  : quagga
group to run as  : quagga
group for vty sockets : 
config file mask        : 0600
log file mask           : 0600

Now, let us try Quagga for some examples

• Zebra
• Copy sample to conf file:
• cd /opt/quagga
• cp zebra.conf.sample zebra.conf
• Start zebra daemon, you use this command 
• sudo /opt/quagga/sbin/zebra &
• Check the daemon zebra:
• netstat -a | grep zebra 
• Get into the zebra, you use 
• telnet localhost zebra
• BGP
• Copy sample to conf file:
• cd /opt/quagga
• cp bgpd.conf.sample bgpd.conf
• Start bgpd: 
• sudo /opt/quagga/sbin/bgpd &
• Check the daemon zebra:
• netstat -a | grep bgpd
• Get into the bgpd, you use 
• telnet localhost bgpd
• vtysh
• cd /opt/quagga
• vi /opt/quagga/vtysh.conf
• sudo /opt/quagga/bin/vtysh

! Sample configuration file for vtysh.
!
!service integrated-vtysh-config
hostname quagga-router
username root nopassword
!

Stop Quagga Daemon ( for example: bgpd ):

• sudo kill `cat /opt/quagga/bgpd.pid`

Port Number:

• zebra: 2601
• ripd:  2602
• ripng: 2603
• ospfd: 2604
• bgpd:  2605
• ospf6d: 2606

Reference:
http://openmaniak.com/quagga_tutorial.php

http://wiki.nil.com/Installing_and_running_Quagga#Initial_configuratiom

http://sourceforge.net/apps/mediawiki/quagga/index.php?title=TipsNTricks

[EVB] What is EVB?

The introduction of EVB
http://networkerhere.blogspot.tw/2012/11/impact-of-server-virtualization-on_13.html
http://blog.ipspace.net/2011/05/edge-virtual-bridging-evb-8021qbg-eases.html

Due to overlay network, EVB is becoming to die ...!? The most main reason is that VLAN is not able to scale:
http://blog.ipspace.net/2012/02/edge-virtual-bridging-8021qbg.html

There are also some similar approaches, such as, VN-Tags that can do the same thing as EVB.
http://www.definethecloud.net/access-layer-network-virtualization-vn-tag-and-vepa/

[Puppet] The studying track of Puppet

After taking some time to study Puppet a while, I become to believe more and more that it is a great auto configuration tool. The following URLs are my studying track of Puppet.

For the beginner, it is recommended to walk through for learning the basic concept
Learning Puppet
http://docs.puppetlabs.com/learning/index.html

Once you finish the above document, you probably want to do your own type and provider. Here you go:
Customer Type
http://docs.puppetlabs.com/guides/custom_types.html
Provider Development
http://docs.puppetlabs.com/guides/provider_development.html

Juniper provides its open source project for using Puppet to manage and configure the switches that support Puppet Agent. It is a very good example for you to understand how to define customer type and to implement provider.

To define customer type:
https://github.com/NetdevOps/puppet-netdev-stdlib
To implement provider:
https://github.com/Juniper/puppet-netdev-stdlib-junos

[OpenFlow 1.X] The Flow Table Usage

When OpenFlow 1.0 guys try to reach the version 1.1 or more, the first question coming up with would most likely be "how to use multi-flow tables ?" Well, we could see an example in RYU OpenFlow Controller as follow: http://www.slideshare.net/yamahata/ryu-sdnframeworkupload
In page 33, there are 3 flow tables which contains match conditons and actions. It can give an initial idea for that question.






P.S: Pica8 works with Broadcom to double flow table size in its OF1.3 Switch
http://searchsdn.techtarget.com/news/2240214709/Pica8-doubles-flow-rule-capacity-in-its-new-OpenFlow-13-switch

[Info] Interface to the Routing System (I2RS)

IETF is trying to build its SDN standard by forming a group called Interface to the Routing System (I2RS) .

They are going to define SDN based on networking vendor side and could effect SDN market sooner or later. 

http://searchsdn.techtarget.com/opinion/IETF-SDN-standards-emerge-Southbound-protocols-NFV-service-chains?src=5174092

The Drafts:

Interface to the Routing System (i2rs) Internet Drafts

[Trema] A good example with Trema to build OpenFlow Controller ( use case )

Previously I just wondered how to use Trema to build a useful and piratical solution with a good architecture for users to leverage its functionality. Well, it does exist. The following URL is a good example of a good use case containing a good architecture.
http://www.slideshare.net/chibayasunobu/developing-production-open-flow-controller-with-trema-201304160

Source Code:
https://github.com/trema/virtual-network-platform


Design:

Implementation:

[Network] EtherTypes Reference

802 EtherTypes Reference:

0x0000 - 0x05DC	IEEE 802.3 length.
0x0600	XEROX NS IDP.
0x0660 0x0661	DLOG.
0x0800	IP, Internet Protocol.
0x0801	X.75 Internet.
0x0802	NBS Internet.
0x0803	ECMA Internet.
0x0804	Chaosnet.
0x0805	X.25 Level 3.
0x0806	ARP, Address Resolution Protocol.
0x0807	XNS compatability.
0x0808	Frame Relay ARP.
0x8035	DRARP, Dynamic RARP. RARP, Reverse Address Resolution Protocol.
0x80F3	AARP, AppleTalk Address Resolution Protocol.
0x8100	EAPS, Ethernet Automatic Protection Switching.
0x8137	IPX, Internet Packet Exchange.
0x814C	SNMP, Simple Network Management Protocol.
0x86DD	IPv6, Internet Protocol version 6.
0x8808	MPCP, Multi-Point Control Protocol.
0x880B	PPP, Point-to-Point Protocol.
0x880C	GSMP, General Switch Management Protocol.
0x8847	MPLS, Multi-Protocol Label Switching (unicast).
0x8848	MPLS, Multi-Protocol Label Switching (multicast).
0x8863	PPPoE, PPP Over Ethernet (Discovery Stage).
0x8864	PPPoE, PPP Over Ethernet (PPP Session Stage).
0x886F	Network Load Balancing.
0x888E	EAPOL, EAP over LAN.
0x88A2	AoE, ATA over Ethernet.
0x88A4	EtherCAT.
0x88CA	TIPC, Transparent Inter Process Communication Protocol.
0x88BB	LWAPP, Light Weight Access Point Protocol.
0x88CC	LLDP, Link Layer Discovery Protocol.
0x88DC	WSMP, WAVE S

[Static Analysis] Tools for static analysis ( C/C++ )

Here are the list of some software tools for static analysis ( C/C++ ). FYI.

• Klocwork Insight ( Commercial )
• Cppcheck
• LintProject Pro
• scan-build

[dbus] The notes for DBus APIs

The previous post: [dbus] The useful information and example of D-Bus has introduced the concept and an example. This post is focused on its API.

DBusConnection:
http://dbus.freedesktop.org/doc/api/html/group__DBusConnection.html#details
The keypoint that is worth mentioning is about dealing with asynchronous message. 
The function dbus_connection_read_write_dispatch() for example does all three of these things, offering a simple alternative to a main loop. If you don't need to be asynchronous, you can ignore DBusWatch, DBusTimeout, and dbus_connection_dispatch(). Instead, dbus_connection_read_write_dispatch() can be used.

Here is an example of how to use dbus watch and dbus timeout:
http://stackoverflow.com/questions/9378593/dbuswatch-and-dbustimeout-examples

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/socket.h>
#include <arpa/inet.h>

#include <dbus/dbus.h>

struct dbus_ctx {
    DBusConnection *conn;
    struct event_base *evbase;
    struct event dispatch_ev;
    void *extra;
};

static void dispatch(int fd, short ev, void *x)
{
    struct dbus_ctx *ctx = x;
    DBusConnection *c = ctx->conn;

    logger(LOG_DEBUG "dispatching\n");

    while (dbus_connection_get_dispatch_status(c) == DBUS_DISPATCH_DATA_REMAINS)
        dbus_connection_dispatch(c);
}

static void handle_dispatch_status(DBusConnection *c,
                                   DBusDispatchStatus status, void *data)
{
    struct dbus_ctx *ctx = data;

    logger(LOG_DEBUG "new dbus dispatch status: %d\n", status);

    if (status == DBUS_DISPATCH_DATA_REMAINS) {
        struct timeval tv = {
            .tv_sec = 0,
            .tv_usec = 0,
        };
        event_add(&ctx->dispatch_ev, &tv);
    }
}

static void handle_watch(int fd, short events, void *x)
{
    struct dbus_ctx *ctx = x;
    struct DBusWatch *watch = ctx->extra;

    unsigned int flags = 0;
    if (events & EV_READ)
        flags |= DBUS_WATCH_READABLE;
    if (events & EV_WRITE)
        flags |= DBUS_WATCH_WRITABLE;
    /*if (events & HUP)
        flags |= DBUS_WATCH_HANGUP;
    if (events & ERR)
        flags |= DBUS_WATCH_ERROR;*/

    logger(LOG_DEBUG "got dbus watch event fd=%d watch=%p ev=%d\n",
           fd, watch, events);
    if (dbus_watch_handle(watch, flags) == FALSE)
        logger(LOG_ERROR "dbus_watch_handle() failed\n");

    handle_dispatch_status(ctx->conn, DBUS_DISPATCH_DATA_REMAINS, ctx);
}

static dbus_bool_t add_watch(DBusWatch *w, void *data)
{
    if (!dbus_watch_get_enabled(w))
        return TRUE;

    struct dbus_ctx *ctx = data;
    ctx->extra = w;

    int fd = dbus_watch_get_unix_fd(w);
    unsigned int flags = dbus_watch_get_flags(w);
    short cond = EV_PERSIST;
    if (flags & DBUS_WATCH_READABLE)
        cond |= EV_READ;
    if (flags & DBUS_WATCH_WRITABLE)
        cond |= EV_WRITE;

    struct event *event = event_new(ctx->evbase, fd, cond, handle_watch, ctx);
    if (!event)
        return FALSE;

    event_add(event, NULL);

    dbus_watch_set_data(w, event, NULL);

    logger(LOG_DEBUG "added dbus watch fd=%d watch=%p cond=%d\n", fd, w, cond);
    return TRUE;
}

static void remove_watch(DBusWatch *w, void *data)
{
    struct event *event = dbus_watch_get_data(w);

    if (event)
        event_free(event);

    dbus_watch_set_data(w, NULL, NULL);

    logger(LOG_DEBUG "removed dbus watch watch=%p\n", w);
}

static void toggle_watch(DBusWatch *w, void *data)
{
    logger(LOG_DEBUG "toggling dbus watch watch=%p\n", w);

    if (dbus_watch_get_enabled(w))
        add_watch(w, data);
    else
        remove_watch(w, data);
}

static void handle_timeout(int fd, short ev, void *x)
{
    struct dbus_ctx *ctx = x;
    DBusTimeout *t = ctx->extra;

    logger(LOG_DEBUG "got dbus handle timeout event %p\n", t);

    dbus_timeout_handle(t);
}

static dbus_bool_t add_timeout(DBusTimeout *t, void *data)
{
    struct dbus_ctx *ctx = data;

    if (!dbus_timeout_get_enabled(t))
        return TRUE;

    logger(LOG_DEBUG "adding timeout %p\n", t);

    struct event *event = event_new(ctx->evbase, -1, EV_TIMEOUT|EV_PERSIST,
                                    handle_timeout, t);
    if (!event) {
        logger(LOG_ERROR "failed to allocate new event for timeout\n");
        return FALSE;
    }

    int ms = dbus_timeout_get_interval(t);
    struct timeval tv = {
        .tv_sec = ms / 1000,
        .tv_usec = (ms % 1000) * 1000,
    };
    event_add(event, &tv);

    dbus_timeout_set_data(t, event, NULL);

    return TRUE;
}

static void remove_timeout(DBusTimeout *t, void *data)
{
    struct event *event = dbus_timeout_get_data(t);

    logger(LOG_DEBUG "removing timeout %p\n", t);

    event_free(event);

    dbus_timeout_set_data(t, NULL, NULL);
}

static void toggle_timeout(DBusTimeout *t, void *data)
{
    logger(LOG_DEBUG "toggling timeout %p\n", t);

    if (dbus_timeout_get_enabled(t))
        add_timeout(t, data);
    else
        remove_timeout(t, data);
}

static DBusHandlerResult handle_nameownerchanged(DBusMessage *message,
                                                 void *data)
{
    struct dbus_ctx *ctx = data;
    char *name, *old, *new;
    if (dbus_message_get_args(message, NULL,
                              DBUS_TYPE_STRING, &name,
                              DBUS_TYPE_STRING, &old,
                              DBUS_TYPE_STRING, &new,
                              DBUS_TYPE_INVALID) == FALSE) {
        logger(LOG_ERROR "spurious NameOwnerChanged signal\n");
        return DBUS_HANDLER_RESULT_NOT_YET_HANDLED;
    }
    logger(LOG_DEBUG "dbus NameOwnerChanged %s -> %s\n", old, new);

    if (new[0] != '\0')
        return DBUS_HANDLER_RESULT_NOT_YET_HANDLED;

    /* XXX handle disconnecting clients */

    return DBUS_HANDLER_RESULT_NOT_YET_HANDLED;
}

static DBusHandlerResult msg_filter(DBusConnection *connection,
                                    DBusMessage *message, void *data)
{
    if (dbus_message_is_signal(message, DBUS_INTERFACE_DBUS,
                               "NameOwnerChanged"))
        return handle_nameownerchanged(message, data);

    logger(LOG_DEBUG "got dbus message %d %s -> %s %s/%s/%s %s\n",
           dbus_message_get_type(message),
           dbus_message_get_sender(message),
           dbus_message_get_destination(message),
           dbus_message_get_path(message),
           dbus_message_get_interface(message),
           dbus_message_get_member(message),
           dbus_message_get_type(message) == DBUS_MESSAGE_TYPE_ERROR ?
           dbus_message_get_error_name(message) : "");

    return DBUS_HANDLER_RESULT_NOT_YET_HANDLED;
}

static void unregister_func(DBusConnection *connection, void *data)
{
}

static DBusHandlerResult message_func(DBusConnection *connection,
                                      DBusMessage *message, void *data)
{
    struct dbus_ctx *ctx = data;

    logger(LOG_DEBUG "got dbus message sent to %s %s %s\n",
           dbus_message_get_destination(message),
           dbus_message_get_interface(message),
           dbus_message_get_path(message));

    /* XXX handle DBus message */

    return DBUS_HANDLER_RESULT_HANDLED;
}

static DBusObjectPathVTable dbus_vtable = {
    .unregister_function = unregister_func,
    .message_function = message_func,
};

struct dbus_ctx *dbus_init(struct event_base *eb)
{
    DBusConnection *conn = NULL;
    struct dbus_ctx *ctx = calloc(1, sizeof(struct dbus_ctx));
    if (!ctx) {
        logger_perror("can't allocate dbus_ctx\n");
        goto out;
    }

    conn = dbus_bus_get_private(DBUS_BUS_SESSION, NULL);
    if (conn == NULL) {
        logger(LOG_ERROR "failed to get bus\n");
        goto out;
    }

    dbus_connection_set_exit_on_disconnect(conn, FALSE);

    ctx->conn = conn;
    ctx->evbase = eb;
    event_assign(&ctx->dispatch_ev, eb, -1, EV_TIMEOUT, dispatch, ctx);

    if (!dbus_connection_set_watch_functions(conn, add_watch, remove_watch,
                                             toggle_watch, ctx, NULL)) {
        logger(LOG_ERROR "dbus_connection_set_watch_functions() failed\n");
        goto out;
    }

    if (!dbus_connection_set_timeout_functions(conn, add_timeout,
                                               remove_timeout, toggle_timeout,
                                               ctx, NULL)) {
        logger(LOG_ERROR "dbus_connection_set_timeout_functions() failed\n");
        goto out;
    }

    if (dbus_connection_add_filter(conn, msg_filter, ctx, NULL) == FALSE) {
        logger(LOG_ERROR "dbus_connection_add_filter() failed\n");
        goto out;
    }

    dbus_connection_set_dispatch_status_function(conn, handle_dispatch_status,
                                                 ctx, NULL);

    char match[256];
    snprintf(match,
             sizeof(match),
             "type='signal',interface='%s',member='NameOwnerChanged'",
             DBUS_INTERFACE_DBUS);
    DBusError error;
    dbus_error_init(&error);
    dbus_bus_add_match(conn, match, &error);
    if (dbus_error_is_set(&error)) {
        logger(LOG_ERROR "dbus_bus_add_match() %s failed: %s\n",
               "NameOwnerChanged", error.message);
        dbus_error_free(&error);
        goto out;
    }

    snprintf(match,
             sizeof(match),
             "type='signal',interface='%s',member='%s'",
             GNP_IPC_INTERFACE, GNP_IPC_SIGNAL_DELIVER_SA);
    dbus_error_init(&error);
    dbus_bus_add_match(conn, match, &error);
    if (dbus_error_is_set(&error)) {
        logger(LOG_ERROR "dbus_bus_add_match() %s failed: %s\n",
               GNP_IPC_SIGNAL_DELIVER_SA, error.message);
        dbus_error_free(&error);
        goto out;
    }

    if (dbus_connection_register_object_path(conn, GNP_IPC_PATH, &dbus_vtable,
                                             ctx) != TRUE) {
        logger(LOG_ERROR "failed to register object path\n");
        goto out;
    }

    return ctx;

out:
    if (conn) {
        dbus_connection_close(conn);
        dbus_connection_unref(conn);
    }
    if (ctx)
        free(ctx);
    return NULL;
}

void dbus_close(struct dbus_ctx *ctx)
{
    if (ctx && ctx->conn) {
        dbus_connection_flush(ctx->conn);
        dbus_connection_close(ctx->conn);
        dbus_connection_unref(ctx->conn);
        event_del(&ctx->dispatch_ev);
    }
    if (ctx)
        free(ctx);
}

DBusMessage:
http://dbus.freedesktop.org/doc/api/html/group__DBusMessage.html
This module is to deal with messages.

DBus Message Bus API:
http://dbus.freedesktop.org/doc/api/html/group__DBusBus.html
This module is the only one in libdbus that's specific to communicating with the message bus daemon. The rest of the API can also be used for connecting to another application directly.

DBusPendingCall API:
http://dbus.freedesktop.org/doc/api/html/group__DBusPendingCall.html


An example to look over names on the message (bus driver) by using proxy object:
http://stackoverflow.com/questions/14263390/how-to-compile-a-basic-d-bus-glib-example

#include <stdlib.h>          // for exit()   
#include <dbus/dbus.h>       // for dbus_*   
#include <dbus/dbus-glib.h>  // for dbus_g_*

int
main (int argc, char **argv)
{
  DBusGConnection *connection;
  GError *error;
  DBusGProxy *proxy;
  char **name_list;
  char **name_list_ptr;

  g_type_init ();

  error = NULL;
  connection = dbus_g_bus_get (DBUS_BUS_SESSION,
                           &error);
  if (connection == NULL)
    {
      g_printerr ("Failed to open connection to bus: %s\n",
              error->message);
      g_error_free (error);
      exit (1);
    }

  /* Create a proxy object for the "bus driver" (name "org.freedesktop.DBus") */

  proxy = dbus_g_proxy_new_for_name (connection,
                                 DBUS_SERVICE_DBUS,
                                 DBUS_PATH_DBUS,
                                 DBUS_INTERFACE_DBUS);

  /* Call ListNames method, wait for reply */
  error = NULL;
  if (!dbus_g_proxy_call (proxy, "ListNames", &error, G_TYPE_INVALID,
                      G_TYPE_STRV, &name_list, G_TYPE_INVALID))
    {
      /* Just do demonstrate remote exceptions versus regular GError */
      if (error->domain == DBUS_GERROR && error->code == DBUS_GERROR_REMOTE_EXCEPTION)
    g_printerr ("Caught remote method exception %s: %s",
            dbus_g_error_get_name (error),
            error->message);
      else
    g_printerr ("Error: %s\n", error->message);
      g_error_free (error);
      exit (1);
    }

  /* Print the results */

  g_print ("Names on the message bus:\n");

  for (name_list_ptr = name_list; *name_list_ptr; name_list_ptr++)
    {
      g_print ("  %s\n", *name_list_ptr);
    }
  g_strfreev (name_list);

  g_object_unref (proxy);

  return 0;
}

The example of dbus_connection_add_filter() and handler method:
http://lists.freedesktop.org/archives/dbus/2003-September/000468.html