[epoll] How to use epoll in socket program?

When we do socket program, the most popular way is of PPC/TPC mode ( Process Per Connection / Thread Per Connection ). But the mode has some drawbacks, for instance, it will consum too much memory and CPU for context switch when creating more than hundreds of connections. epoll is good solution to deal with that.

The following URL is a good explaintion about epoll.
How to use epoll? A complete example in C

And here is a complete examle of using epoll in socket program.

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#define SERVPORT 9527 /* Server Port Number */
#define BACKLOG 10 /* The Max of Client Number */

void setnonblocking(int sock)
{
    int opts;
    opts = fcntl(sock,F_GETFL);
    if(opts<0)
    {
        perror("fcntl(sock,GETFL)");
        exit(1);
    }
    opts = opts|O_NONBLOCK;
    if(fcntl(sock,F_SETFL,opts)<0)
    {
        perror("fcntl(sock,SETFL,opts)");
        exit(1);
    }  
}

void do_use_fd(int client_fd)
{
    const char str[] = "God bless you!\n";
    if (send(client_fd,  str,  sizeof(str),  0) == -1)
        perror("send");
    close(client_fd);
}

int main()
{
    int sockfd;
    struct sockaddr_in my_addr;
    struct sockaddr_in remote_addr;
    if ((sockfd = socket(AF_INET,  SOCK_STREAM,  0)) == -1) {
        perror("socket");
        exit(1);
    }
    my_addr.sin_family = AF_INET;
    my_addr.sin_port = htons(SERVPORT);
    my_addr.sin_addr.s_addr = INADDR_ANY;
    bzero(&(my_addr.sin_zero), 8);
    if (bind(sockfd,  (struct sockaddr *)&my_addr,  sizeof(struct sockaddr)) == -1) {
        perror("bind");
        exit(1);
    }
    printf("bind ok\n");
    if (listen(sockfd,  BACKLOG) == -1) {
        perror("listen");
        exit(1);
    }
    printf("listen ok\b");
    size_t sin_size = sizeof(struct sockaddr_in);

#define MAX_EVENTS 10
    struct epoll_event ev, events[MAX_EVENTS];
    int conn_sock, nfds, epollfd;

    epollfd = epoll_create(10);
    if (epollfd == -1) {
        perror("epoll_create");
        exit(EXIT_FAILURE);
    }
    printf("epoll_create\n");

    ev.events = EPOLLIN;
    ev.data.fd = sockfd;
    if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &ev) == -1) {
        perror("epoll_ctl: sockfd");
        exit(EXIT_FAILURE);
    }
    printf("epoll_ctl ok\n");

    for (;;) {
        printf("start epoll_wait\n");
        nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
        if (nfds == -1) {
            perror("epoll_pwait");
            exit(EXIT_FAILURE);
        }
        printf("epoll_wait returns, nfds = %d\n", nfds);

        for (int n = 0; n < nfds; ++n) {
            if (events[n].data.fd == sockfd) {
                conn_sock = accept(sockfd,
                        (struct sockaddr *) &remote_addr, &sin_size);
                if (conn_sock == -1) {
                    perror("accept");
                    exit(EXIT_FAILURE);
                }
                setnonblocking(conn_sock);
                ev.events = EPOLLIN | EPOLLET;
                ev.data.fd = conn_sock;
                if (epoll_ctl(epollfd, EPOLL_CTL_ADD, conn_sock,
                            &ev) == -1) {
                    perror("epoll_ctl: conn_sock");
                    exit(EXIT_FAILURE);
                }
            } else {
                do_use_fd(events[n].data.fd);
            }
        }
    }
}

[Socket] How to use he function fcntl()

When server is running to accept() and no client connection request comes, what it happens? When the server stops waiting for the arrival of the connection request on accept(). Similarly, when the program is running to receive(), ifno data can be read, the program will also stop receiving statements. This situation is called blocking. 
If you want to only pay attention to check whether there are customers waiting for a connection server to accept connection, otherwise the program continues to do other things, through the Socket set for non-blocking way: non-blocking socket when no customers waiting for accept calls return immediately.
 
# include <unistd.h>
# include <fcntl.h>
....
sockfd = socket (AF_INET, SOCK_STREAM, 0);
fcntl (sockfd, F_SETFL, O_NONBLOCK);
....
 
Set socket to non-blocking mode, "polling" the Socket. When no data is waiting to be processed from a non-blocking Socket ( for read data ), the function will return immediately, and the return value is set to -1 and errno set of EWOULDBLOCK. 
But this "polling" causes the CPU is busy waiting mode, thereby reducing performance.


P.S:
There is another way to deal with non-blocking socket.
ioctl(sockfd, FIONBIO, 0); // 0 is about non-blocking mode

[Storage] Some articles of NoSQL and Object Storage ( S3 and Swift )

Severl months ago, I saw the first article "LINE Starage" and didn't pay too much attention on it. Recently Someone discussed storage with me and let me remember it. So I found it again ( took me a bunch of time to look for ) and also took some pretty nice related articles here as well.

LINE Storage: Storing billions of rows in Sharded-Redis and HBase per Month

http://tech.naver.jp/blog/?p=1420

NoSQL Data Modeling Techniques

http://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/

Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
 

Hadoop + S3

http://wiki.apache.org/hadoop/AmazonS3

Apache Hadoop over OpenStack Swift

http://bigdatacraft.com/archives/349

[Sqlite] A simple example of Sqlite in C

Here is a simple example of how to deal with Sqlite Database in C language.

#include <stdio.h> #include <sqlite3.h> static char *createsql = "CREATE TABLE Employee(" "ID INTEGER PRIMARY KEY," "Name VARCHAR(10)," "BadgeID VARCHAR(10));"; static char *insertsql = "INSERT INTO Employee VALUES(NULL, 'Danny', '12345');"; static char *querysql = "SELECT * FROM Employee;"; void main(void) { int rows, cols; sqlite3 *db; char *errMsg = NULL; char **result; /* Open database file */ if (sqlite3_open("my_example.db3", &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL)) { return; } /* Build Table */ sqlite3_exec(db, createsql, 0, 0, &errMsg); /* Add a new record */ sqlite3_exec(db, insertsql, 0, 0, &errMsg); /* Get the last insert record's ID */ printf("%d\n", sqlite3_last_insert_rowid(db)); /* Get all records in database */ sqlite3_get_table(db , querysql, &result , &rows, &cols, &errMsg); /* List all the data */ for (i=0;i<rows;i++) { for (j=0;j<cols;j++) { printf("%s\t", result[i*cols+j]); } printf("\n"); } /* Free table */ sqlite3_free_table(result); /* Close database */ sqlite3_close(db); }

When Compiling, just add sqlite's flag  as follows:

gcc -lsqlite3 -o sqlite_exp sqlite_exp.c

[tun/tap] An introduction of TUN/TAP

http://www.kernel.org/pub/linux/kernel/people/marcelo/linux-2.4/Documentation/networking/tuntap.txt

TUN/TAP provides packet reception and transmission for user space programs. 
It can be viewed as a simple Point-to-Point or Ethernet device, which 
instead of receiving packets from a physical media, receives them from 
user space program and instead of sending packets via physical media 
writes them to the user space program. 

When a program opens /dev/net/tun, driver creates and registers corresponding
net device tunX or tapX. After a program closed above devices, driver will 
automatically delete tunX or tapX device and all routes corresponding to it.

And also, here is a web site about Universal TUN/TAP device driver Frequently Asked Question.

http://vtun.sourceforge.net/tun/faq.html

The description of Universal TUN/TAP device driver

https://www.kernel.org/doc/Documentation/networking/tuntap.txt

[SLB] Server Load Balancing with DSR

IPVS ( based on LVS) provides 3 kind of ways to do server load balancing:

1. Virtual Server via Network Address Translation（VS/NAT）
2. Virtual Server via IP Tunneling（VS/TUN）
3. Virtual Server via Direct Routing（VS/DR） 

Recently I find another way to do load balancing, and it is Direct Server Return
http://techiess.wordpress.com/2010/09/09/load-balancing-dsr-direct-server-return/
Using DSR, packet is returning directly from the server to client without going through L4 switch.

[Virtual Network] Midokura’s MidoNet

Recently I saw the article about Midokura MidoNet as follows:
http://blog.ioshints.info/2012/08/midokuras-midonet-layer-2-4-virtual.html
It is a layer 2-4 virtual network solution and catches my attention and lets me want to know more about.

Meanwhile, Midokura also provide Quantum Plugin for MidoNet to integrate into OpenStack.
https://github.com/midokura/midonet-openstack

On its web site, it provides a lot of features in virtual network solution and is pretty amazing. Due to MidoNet is not an open source project, so we are not able to see how it exactly achieves all of these things.
http://www.midokura.com/midonet/openstack/

MidoNet Key Features

• Fully virtualized Layer 2 through 7 networking
• VLAN-less VLANs – Virtual L2 Distributed Level Isolation and Switching with virtually no limitations
• Fully Distributed Architecture with no single points of failure
• Virtual L3 Distributed Routing
• Distributed Load Balancing and Firewall Services
• NAT
• VPN
• Access Control Lists (ACLs)
• Restful API
• Web Based Management Control Panel
• Monitoring and Metering of Networking Services
• Live Migration
• Rate Limiting
• Virtual Tap (Port Monitoring and Mirroring)

[OpenStack] Existing High Availability Options for Networking

 Existing High Availability Options for Networking

    OpenStack offical web site has document to explain the existing HA options for networking. I think most of important item is the option 1: multi-host networking mode. This mode avoids from single point of failure when Nova-Network component is down and also eliminate the potential network bottleneck in single-host networking mode.

For more in details, please see the this:
http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html