Danny's tech notebook | 丹尼技術手札

Monday, October 29, 2012

[VirtualBox] Daily Operation in Linux VM ( VDI file ) with tip and trick

If you use VirtulBox on Windows for your virtual machine environment, you will encounter some daily operation issues that are kind of tedious. Here I want to provide some useful commands for quickly solving some common operation in Linux VM.

When you want to duplicate original VDI file to use in other VirtualBox environment, you need to execute this:

> "C:\Program Files\Oracle\VirtualBox\VBoxManage" clonevdi source.vdi output.vdi

To clean up the empty space in your VDI file ( it could reduce the size of your VDI file ), there are 2 steps:

On Linux VM
> dd if=/dev/zero of=fillfile bs=1M ( running on
> rm fillfile
On VirtualBox Host ( Windows )
"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd your_VDI.vdi --compact

To change the size of your VirtualBox VDI file

"C:\Program Files\Oracle\VirtualBox\VBoxManage"--resize SIZE_IN_MB
YOUR_HARD_DISK.vdi

Tuesday, October 9, 2012

[Trema] How to sync threads using ZeroMQ in Trema

If someone is familiar with Trema, then he/she knows that the socket mode in Trema is non-blocking, which means there is no waiting when sending the request message. But, here is an question. If we want to use ZeroMQ ( in different thread ) to receive the flow stats request from outside world and how do we sync threads and get flow stats info from OFSwitch?

My approach uses "pthread mutex". The key point is in the red box as following picture. When Zmq_Responder receives flow stats request, it will run a callback function, lock the mutex variable, and send flow stats request to OF Switch ( it is non-blocking socket mode ). After sending, Zmq_Responder will be locked and will wait for OF Switch to reply flow stats message. Once it replies, Routing Switch in the main thread will produce the reply message and unlock the mutex variable and let Zmq_Responder to reply to outside world.

The approach can sync these two threads in the sequence, but also has some kind of dead-lock risk. If some one has a better solution, let me know if you are willing.

Tuesday, October 2, 2012

[Trema] Try the cucumber scenarios in Trema

When you install Trema ready, you can try this in Trema path:
trema> sudo cucumber -r features

And then you will get a lot of log and info on screen. For instance, this is what I got:

Failing Scenarios:
cucumber features/example.message.echo_reply.feature:8 # Scenario: Send echo reply x 10
cucumber features/example.message.echo_reply.feature:18 # Scenario: Send echo reply x 10 in Ruby
cucumber features/example.message.echo_request.feature:8 # Scenario: Send echo request x 10
cucumber features/example.message.echo_request.feature:18 # Scenario: Send echo request x 10 in Ruby
cucumber features/example.message.features_request.feature:8 # Scenario: Send a features request
cucumber features/example.message.features_request.feature:48 # Scenario: Send a features request in Ruby
cucumber features/example.message.hello.feature:8 # Scenario: Hello trema
cucumber features/example.message.hello.feature:18 # Scenario: Hello trema in Ruby
cucumber features/example.message.set_config.feature:8 # Scenario: set config x 10
cucumber features/example.message.set_config.feature:19 # Scenario: set config x 10 in Ruby
cucumber features/example.packetin_filter_config.feature:8 # Scenario: add filter
cucumber features/example.packetin_filter_config.feature:23 # Scenario: dump filter
cucumber features/example.packetin_filter_config.feature:40 # Scenario: dump filter strict
cucumber features/example.packetin_filter_config.feature:62 # Scenario: delete filter strict
cucumber features/example.packetin_filter_config.feature:83 # Scenario: delete filter
cucumber features/packetin_filter.feature:8 # Scenario: packetin_filter --help
cucumber features/packetin_filter.feature:33 # Scenario: packetin_filter -h
cucumber features/switch_manager.feature:8 # Scenario: switch_manager --help
cucumber features/switch_manager.feature:26 # Scenario: switch_manager -h

84 scenarios (19 failed, 65 passed)

417 steps (19 failed, 47 skipped, 351 passed)

7m0.821s

Or if you want to further test Trema/Apps, for instance, sliceable_switch app, you can do this:
trema> sudo cucumber -r features -r ../apps/sliceable_switch/features /home/liudanny/SourceCode/apps/sliceable_switch/features/port_binding.feature
But, I am not sure it is correct way or not and will figure it out.

[Cucumber] how to use cucumber with a simple example

As my early post [Cucumber] An brief introduction, Cucumber is an amazing testing tool and it is about Behaviour-Driven Development instead of Test-Driven Development.

The book "The Cucumber Book" gives a very good explanation and useful of examles to illustrate the usages and functionalities in depths.

Based on that book's a good simple example, the following picture give the dictionary structure (left hand side ) and 3 of important files ( right hand side ) as follows:

"adding.feature" is file contained scenario ( test case )
"calc.rb" provides the calculation function for calculator_steps.rb
" calculator_steps.rb" provide the functions mapping to feature's script based on the key words: Given, When, and Then

Under the path, we run "cucumber" and then get the information as below:

Feature: Adding

Scenario Outline: Add two numbers      # features/adding.feature:3
    Given the input "<input>"            # features/step_definitions/calculator_steps.rb:9
    When the calculator is run           # features/step_definitions/calculator_steps.rb:13
    Then the output should be "<output>" # features/step_definitions/calculator_steps.rb:18

    Examples:
      | input | output |
      | 2+2    | 4    |
      | 98+1 | 99      |

2 scenarios (2 passed)
6 steps (6 passed)
0m0.012s

Wednesday, September 26, 2012

[JSON] How to use jansson lib to generate JSON data in C

This exmple method will generate the result string below:

{ "command":"link_status",
"result":["AAABBBCCC"] }

char* 
generate_json_data() {
  
  char *ret_strings = NULL;
  char *ret_string = "AAABBBCCC";
  
  json_t *root = json_object();
  json_t *result_json_arr = json_array();
  
  json_object_set_new( root, "command", json_string( "link_status" ) );
  json_object_set_new( root, "result", result_json_arr );
  
  ...
    
  json_array_append( result_json_arr, json_string( ret_string ) );
  free( ret_string );
  ret_strings = json_dumps( root, 0 );
  json_decref( root );
  return ret_strings;
}


For more example in details: 

#include <stdio.h>
#include <jansson.h>

void add_2array_to_json( json_t* obj, const char* name, const int*
marr, size_t dim1, size_t dim2 )
{
    size_t i, j;
    json_t* jarr1 = json_array();

    for( i=0; i<dim1; ++i ) {
        json_t* jarr2 = json_array();

        for( j=0; j<dim2; ++j ) {
            int val = marr[ i*dim2 + j ];
            json_t* jval = json_integer( val );
            json_array_append_new( jarr2, jval );
        }
        json_array_append_new( jarr1, jarr2 );
    }
    json_object_set_new( obj, name, jarr1 );
    return;
}

int main()
{
    json_t* jdata;
    char* s;
    int arr1[2][3] = { {1,2,3}, {4,5,6} };
    int arr2[4][4] = { {1,2,3,4}, {5,6,7,8}, {9,10,11,12}, {13,14,15,16} };

    jdata = json_object();
    add_2array_to_json( jdata, "arr1", &arr1[0][0], 2, 3 );
    add_2array_to_json( jdata, "arr2", &arr2[0][0], 4, 4 );

    s = json_dumps( jdata, 0 );
    puts( s );
    free( s );
    json_decref( jdata );

    return 0;
}


When run it outputs:
{"arr1": [[1, 2, 3], [4, 5, 6]], "arr2": [[1, 2, 3, 4], [5, 6, 7, 8],
[9, 10, 11, 12], [13, 14, 15, 16]]}

Another example:

#include "jansson.h"  
   
 ...  
   
 json_error_t error;  
 json_t *root = json_loads( reply->data.buffer, 0, &error );  
 if( root )  
 {  
   json_t *jsonData = json_object_get( root, "data" );  
   if( json_is_array( jsonData ) )  
   {  
     const uint length = json_array_size( jsonData );  
     for( uint i=0; i<length; ++i ) // Iterates over the sequence elements.  
     {  
       json_t *jsonObject = json_array_get( jsonData, i );  
         
       json_t *jsonID = json_object_get( jsonObject, "id" );  
       const char *jsonStringID = json_string_value( jsonID );  
         
       json_t *jsonName = json_object_get( jsonObject, "name" );  
       const char *jsonStringName = json_string_value( jsonName );  
         
       // We can now do something with our Name and ID  
     }  
   }  
   json_decref( root ); 
 }  

[epoll] How to use epoll in socket program?

When we do socket program, the most popular way is of PPC/TPC mode ( Process Per Connection / Thread Per Connection ). But the mode has some drawbacks, for instance, it will consum too much memory and CPU for context switch when creating more than hundreds of connections. epoll is good solution to deal with that.

The following URL is a good explaintion about epoll.
How to use epoll? A complete example in C

And here is a complete examle of using epoll in socket program.

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#define SERVPORT 9527 /* Server Port Number */
#define BACKLOG 10 /* The Max of Client Number */

void setnonblocking(int sock)
{
    int opts;
    opts = fcntl(sock,F_GETFL);
    if(opts<0)
    {
        perror("fcntl(sock,GETFL)");
        exit(1);
    }
    opts = opts|O_NONBLOCK;
    if(fcntl(sock,F_SETFL,opts)<0)
    {
        perror("fcntl(sock,SETFL,opts)");
        exit(1);
    }  
}

void do_use_fd(int client_fd)
{
    const char str[] = "God bless you!\n";
    if (send(client_fd,  str,  sizeof(str),  0) == -1)
        perror("send");
    close(client_fd);
}

int main()
{
    int sockfd;
    struct sockaddr_in my_addr;
    struct sockaddr_in remote_addr;
    if ((sockfd = socket(AF_INET,  SOCK_STREAM,  0)) == -1) {
        perror("socket");
        exit(1);
    }
    my_addr.sin_family = AF_INET;
    my_addr.sin_port = htons(SERVPORT);
    my_addr.sin_addr.s_addr = INADDR_ANY;
    bzero(&(my_addr.sin_zero), 8);
    if (bind(sockfd,  (struct sockaddr *)&my_addr,  sizeof(struct sockaddr)) == -1) {
        perror("bind");
        exit(1);
    }
    printf("bind ok\n");
    if (listen(sockfd,  BACKLOG) == -1) {
        perror("listen");
        exit(1);
    }
    printf("listen ok\b");
    size_t sin_size = sizeof(struct sockaddr_in);

#define MAX_EVENTS 10
    struct epoll_event ev, events[MAX_EVENTS];
    int conn_sock, nfds, epollfd;

    epollfd = epoll_create(10);
    if (epollfd == -1) {
        perror("epoll_create");
        exit(EXIT_FAILURE);
    }
    printf("epoll_create\n");

    ev.events = EPOLLIN;
    ev.data.fd = sockfd;
    if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &ev) == -1) {
        perror("epoll_ctl: sockfd");
        exit(EXIT_FAILURE);
    }
    printf("epoll_ctl ok\n");

    for (;;) {
        printf("start epoll_wait\n");
        nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
        if (nfds == -1) {
            perror("epoll_pwait");
            exit(EXIT_FAILURE);
        }
        printf("epoll_wait returns, nfds = %d\n", nfds);

        for (int n = 0; n < nfds; ++n) {
            if (events[n].data.fd == sockfd) {
                conn_sock = accept(sockfd,
                        (struct sockaddr *) &remote_addr, &sin_size);
                if (conn_sock == -1) {
                    perror("accept");
                    exit(EXIT_FAILURE);
                }
                setnonblocking(conn_sock);
                ev.events = EPOLLIN | EPOLLET;
                ev.data.fd = conn_sock;
                if (epoll_ctl(epollfd, EPOLL_CTL_ADD, conn_sock,
                            &ev) == -1) {
                    perror("epoll_ctl: conn_sock");
                    exit(EXIT_FAILURE);
                }
            } else {
                do_use_fd(events[n].data.fd);
            }
        }
    }
}

Thursday, September 20, 2012

[Socket] How to use he function fcntl()

When server is running to accept() and no client connection request comes, what it happens? When the server stops waiting for the arrival of the connection request on accept(). Similarly, when the program is running to receive(), ifno data can be read, the program will also stop receiving statements. This situation is called blocking.
If you want to only pay attention to check whether there are customers waiting for a connection server to accept connection, otherwise the program continues to do other things, through the Socket set for non-blocking way: non-blocking socket when no customers waiting for accept calls return immediately.

# include <unistd.h>
# include <fcntl.h>
....
sockfd = socket (AF_INET, SOCK_STREAM, 0);
fcntl (sockfd, F_SETFL, O_NONBLOCK);
....

Set socket to non-blocking mode, "polling" the Socket. When no data is waiting to be processed from a non-blocking Socket ( for read data ), the function will return immediately, and the return value is set to -1 and errno set of EWOULDBLOCK.
But this "polling" causes the CPU is busy waiting mode, thereby reducing performance.

P.S:
There is another way to deal with non-blocking socket.
ioctl(sockfd, FIONBIO, 0); // 0 is about non-blocking mode

Friday, September 14, 2012

[Storage] Some articles of NoSQL and Object Storage ( S3 and Swift )

Severl months ago, I saw the first article "LINE Starage" and didn't pay too much attention on it. Recently Someone discussed storage with me and let me remember it. So I found it again ( took me a bunch of time to look for ) and also took some pretty nice related articles here as well.