[Storage] Some articles of NoSQL and Object Storage ( S3 and Swift )

Severl months ago, I saw the first article "LINE Starage" and didn't pay too much attention on it. Recently Someone discussed storage with me and let me remember it. So I found it again ( took me a bunch of time to look for ) and also took some pretty nice related articles here as well.

LINE Storage: Storing billions of rows in Sharded-Redis and HBase per Month

http://tech.naver.jp/blog/?p=1420

NoSQL Data Modeling Techniques

http://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/

Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Membase vs Neo4j comparison

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
 

Hadoop + S3

http://wiki.apache.org/hadoop/AmazonS3

Apache Hadoop over OpenStack Swift

http://bigdatacraft.com/archives/349

[Sqlite] A simple example of Sqlite in C

Here is a simple example of how to deal with Sqlite Database in C language.

#include <stdio.h> #include <sqlite3.h> static char *createsql = "CREATE TABLE Employee(" "ID INTEGER PRIMARY KEY," "Name VARCHAR(10)," "BadgeID VARCHAR(10));"; static char *insertsql = "INSERT INTO Employee VALUES(NULL, 'Danny', '12345');"; static char *querysql = "SELECT * FROM Employee;"; void main(void) { int rows, cols; sqlite3 *db; char *errMsg = NULL; char **result; /* Open database file */ if (sqlite3_open("my_example.db3", &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL)) { return; } /* Build Table */ sqlite3_exec(db, createsql, 0, 0, &errMsg); /* Add a new record */ sqlite3_exec(db, insertsql, 0, 0, &errMsg); /* Get the last insert record's ID */ printf("%d\n", sqlite3_last_insert_rowid(db)); /* Get all records in database */ sqlite3_get_table(db , querysql, &result , &rows, &cols, &errMsg); /* List all the data */ for (i=0;i<rows;i++) { for (j=0;j<cols;j++) { printf("%s\t", result[i*cols+j]); } printf("\n"); } /* Free table */ sqlite3_free_table(result); /* Close database */ sqlite3_close(db); }

When Compiling, just add sqlite's flag  as follows:

gcc -lsqlite3 -o sqlite_exp sqlite_exp.c

[tun/tap] An introduction of TUN/TAP

http://www.kernel.org/pub/linux/kernel/people/marcelo/linux-2.4/Documentation/networking/tuntap.txt

TUN/TAP provides packet reception and transmission for user space programs. 
It can be viewed as a simple Point-to-Point or Ethernet device, which 
instead of receiving packets from a physical media, receives them from 
user space program and instead of sending packets via physical media 
writes them to the user space program. 

When a program opens /dev/net/tun, driver creates and registers corresponding
net device tunX or tapX. After a program closed above devices, driver will 
automatically delete tunX or tapX device and all routes corresponding to it.

And also, here is a web site about Universal TUN/TAP device driver Frequently Asked Question.

http://vtun.sourceforge.net/tun/faq.html

The description of Universal TUN/TAP device driver

https://www.kernel.org/doc/Documentation/networking/tuntap.txt

[SLB] Server Load Balancing with DSR

IPVS ( based on LVS) provides 3 kind of ways to do server load balancing:

1. Virtual Server via Network Address Translation（VS/NAT）
2. Virtual Server via IP Tunneling（VS/TUN）
3. Virtual Server via Direct Routing（VS/DR） 

Recently I find another way to do load balancing, and it is Direct Server Return
http://techiess.wordpress.com/2010/09/09/load-balancing-dsr-direct-server-return/
Using DSR, packet is returning directly from the server to client without going through L4 switch.

[Virtual Network] Midokura’s MidoNet

Recently I saw the article about Midokura MidoNet as follows:
http://blog.ioshints.info/2012/08/midokuras-midonet-layer-2-4-virtual.html
It is a layer 2-4 virtual network solution and catches my attention and lets me want to know more about.

Meanwhile, Midokura also provide Quantum Plugin for MidoNet to integrate into OpenStack.
https://github.com/midokura/midonet-openstack

On its web site, it provides a lot of features in virtual network solution and is pretty amazing. Due to MidoNet is not an open source project, so we are not able to see how it exactly achieves all of these things.
http://www.midokura.com/midonet/openstack/

MidoNet Key Features

• Fully virtualized Layer 2 through 7 networking
• VLAN-less VLANs – Virtual L2 Distributed Level Isolation and Switching with virtually no limitations
• Fully Distributed Architecture with no single points of failure
• Virtual L3 Distributed Routing
• Distributed Load Balancing and Firewall Services
• NAT
• VPN
• Access Control Lists (ACLs)
• Restful API
• Web Based Management Control Panel
• Monitoring and Metering of Networking Services
• Live Migration
• Rate Limiting
• Virtual Tap (Port Monitoring and Mirroring)

[OpenStack] Existing High Availability Options for Networking

 Existing High Availability Options for Networking

    OpenStack offical web site has document to explain the existing HA options for networking. I think most of important item is the option 1: multi-host networking mode. This mode avoids from single point of failure when Nova-Network component is down and also eliminate the potential network bottleneck in single-host networking mode.

For more in details, please see the this:
http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html

[Ubuntu] How to build local apt repositories

Reference : http://odzangba.wordpress.com/2006/10/13/how-to-build-local-apt-repositories/

Summary:

1. Edit your sources.list file 
2. Copy your debian packages to the folder where is the packages repository directory  (*.deb folder) 
3. Change into the packages repository directory  (cd ~/repository) 
4. Generate a Packages.gz file  (sudo dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz) 
5. Install the build-essential package  (sudo apt-get install build-essentiall
6.  Update apt-get server packages list  (sudo apt-get update)

[SDN] HP's SDN solution

http://www.infoworld.com/d/networking/hp-aims-three-part-effort-network-virtualization-200048?source=IFWNLE_nlt_virtualization_2012-08-16

Summary:
HP provides 3 tools (softwares) to deal with network virtualization using SDN

• EVI (Ethernet Virtual Interconnect)
• EVI creates a tunnel through the Layer 3 network by encapsulating the packets traveling between the data centers. Rival Cisco already has software that can do this, called OTV (Overlay Transport Virtualization), but it charges extra for that software.

• MDC (Multitenant Device Context)

• It allows for segregating the resources of multiple tenants in a virtualized environment without buying separate switches. This secures the data and applications of one department or cloud service customer from other tenants.

• VSA

• StoreVirtual VSA, for virtualizing storage management. The software is based on VSA (virtual storage appliance) technology from LeftHand Networks, which HP acquired in 2008.