[Python] Problem with Python logging RotatingFileHandler in Django website

If seeing the log files are not rotated properly or correctly in Django web site, you most likely encounter the problem as the following article described:

Problem with Python logging RotatingFileHandler in Django website

"The log is done via RotatingFileHandler which is configured with 10 log files, 1000000 byte each. The log system works, but this are the log files I get:

-rw-r--r-- 1 apache      apache          83 Jul 23 13:30 hr.log
-rw-r--r-- 1 apache      apache      446276 Jul 23 13:03 hr.log.1
-rw-r--r-- 1 apache      apache      999910 Jul 23 06:00 hr.log.10
-rw-r--r-- 1 apache      apache         415 Jul 23 16:24 hr.log.2
-rw-r--r-- 1 apache      apache      479636 Jul 23 16:03 hr.log.3
-rw-r--r-- 1 apache      apache         710 Jul 23 15:30 hr.log.4
-rw-r--r-- 1 apache      apache      892179 Jul 23 15:03 hr.log.5
-rw-r--r-- 1 apache      apache         166 Jul 23 14:30 hr.log.6
-rw-r--r-- 1 apache      apache      890769 Jul 23 14:03 hr.log.7
-rw-r--r-- 1 apache      apache      999977 Jul 23 12:30 hr.log.8
-rw-r--r-- 1 apache      apache      999961 Jul 23 08:01 hr.log.9

As you can see it is a mess. Last log has been written to file hr.log.2 (Jul 23 16:24) instead of hr.log"

I did some survey and found the root cause is in here:
RotatingFileHandler bugs/errors and a general logging question
"The logging system is thread-safe but not safe
against multiple processes (separate Python instances) writing to the
same file. It certainly sounds like you need a scalable solution - and
having each script send the events to a network logging server seems a
good way of handling the scalability requirement. "

    These words above remind me a lot about the importance of synchronization when using multi-threading and multi-process. Also, scaling is another important item that a lot of people don't care. I want to highlight that we should be vigilant about these.

So, it is not your fault. Don't blame yourself. ( Sorry, I am kidding you ... )
Here is one solution for this issue. Please check out the following link ( it's written in Chinese ):
http://www.djangochina.cn/forum.php?mod=viewthread&tid=118752

[Linux Bonding] 802.3ad bond interface has shown RX dropped packets

If someone uses Linux Bonding and finds some or a lot of RX dropped packets in bond interface, please ignore these dropped packet message because of the following informations:

1. Linux Bonding and Single Nic with 1GE switch are no difference in packet loss
I use iperf tool with UDP to test packet drop and jitter and the result shows that there are no difference of packet loss between linux bonding and single nic.




2. Bond0 RX packet dropped is not a bug.
Please check out this: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/1041070

This is related to the bonding mode and _not_ a bug. The bonding module will drop duplicate frames received on inactive ports, which is normal behavior. [0] Overall the packets should be getting into the machine without problems since they are received on the active slave. To confirm this do the following

1) Check dropped packets from all interfaces. So if eth0/eth1 are connected to bond0, we may see dropped packets for bond0 and eth0, but not for eth1. This depends on which interface is the active interface. This can be checked using the following:
cat /sys/class/net/bond0/bonding/active_slave

So if the active_slave isn't dropping packets, and the inactive slave is dropping packets this is normal in 'active-backup' mode (or any mode where there is an inactive slave).

2) If we want both interfaces to not drop packets we can use 'all_slaves_active' bonding module parameter [0].
Check:
cat /sys/class/net/bond0/bonding/all_slaves_active, it should default to 0 which means drop frames on the inactive slave.

If we set this to 1, we will no longer drop frames:
echo 1 | sudo tee /sys/class/net/bond0/bonding/all_slaves_active

3. This article suggests to turn off rp_filter ( could reduce RX dropped )
https://platform9.com/support/neutron-prerequisites-linuxkvm-overlays-vxlangre-vlans/

echo net.ipv4.conf.all.rp_filter=0 >> /etc/sysctl.conf
echo net.ipv4.conf.default.rp_filter=0 >> /etc/sysctl.conf
echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysctl.conf
echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
sysctl -p

What is reverse path filtering?

Reverse path filtering is a mechanism adopted by the Linux kernel, as well as most of the networking devices out there to check whether a receiving packet source address is routable.

So in other words, when a machine with reverse path filtering enabled recieves a packet, the machine will first check whether the source of the recived packet is reachable through the interface it came in.

If it is routable through the interface which it came, then the machine will accept the packet
If it is not routable through the interface, which it came, then the machine will drop that packet.


Other References:
https://bugs.launchpad.net/fuel/+bug/1471647
https://bugs.launchpad.net/fuel/+bug/1539586

[Ansible] My first step to use Ansible

Before get started to use Ansible, you need to add public ssh key to your remote Server first. If you want to setup SSH keys to allow logging in without a password, you can do so with a single command.
The first thing you’ll need to do is make sure you’ve run the keygen command to generate the keys:

ssh-keygen -t rsa

Then use this command to push the key to the remote server, modifying it to match your server name.

cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'

So, from now on you are able to try Ansible to control your remote server.

# sudo pip install ansible
# sudo mkdir /etc/ansible
# cd /etc/ansible/
# vim hosts
  ==> [my_vm]
          2 10.14.1.106

# ansible my_vm --private-key=/home/liudanny/.ssh/id_rsa --user=ubuntu -m ping
or 
# ansible my_vm -m ping --user ubuntu
10.14.1.106 | success >> {
    "changed": false,
    "ping": "pong"
}
ansible my_vm --user=ubuntu -a "/bin/echo hello"
10.14.1.106 | success | rc=0 >>
hello

If the above steps work fine, we can follow this document to create an instance and check services on OpenStack. Here you go:
http://superuser.openstack.org/articles/using-ansible-for-continuous-integration-on-openstack

Reference:
OpenStack-Ansible Installation Guide
http://docs.openstack.org/developer/openstack-ansible/install-guide/index.html

http://www.yet.org/2014/07/ansible/

http://rundeck.org/

[LBaaS] The Load Balance as a Service trace records

A couple of days ago, my colleague imparts Load Balance as a Service (LBaaS) which is the Neutron Plugin to provide the load balancer functionality in OpenStack. Unavoidably, I still like to drill down how it works so that we won't only understand the surface of this function. This article is only focused on the trace record because I have studied the concept of LBaaS. For those who don't know about its concept and implementation, please check out other resources first, ex: https://wiki.openstack.org/wiki/Neutron/LBaaS/Glossary 

• If created a lb pool ready, you can see something like the following picture. My point is to trace subnet and network port.

• From the "subnet" link, we can trace back to the its detail and also can go to its network detail by clicking the link of network id.  

• Here we can find the vip port that is for our load balancer as follows.

Click it to see its details.

• Now, we will use the first part of port id (70081ac2) to trace what happens in linux network space and tun/tap interface.

• LBaaS agent will create a linux network space and the naming rule is "qlbaas-" with the pool's id. 

# ip netns exec qlbaas-13185f35-3f75-47e7-9fd7-301be7b28e88 ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tap70081ac2-6f Link encap:Ethernet  HWaddr fa:16:3e:16:c7:69
          inet addr:192.168.111.60  Bcast:192.168.111.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe16:c769/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15963 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15762 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:958766 (958.7 KB)  TX bytes:1060728 (1.0 MB)

# ip netns exec qlbaas-13185f35-3f75-47e7-9fd7-301be7b28e88 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.111.1   0.0.0.0         UG    0      0        0 tap70081ac2-6f
192.168.111.0   0.0.0.0         255.255.255.0   U     0      0        0 tap70081ac2-6f

• The tap interface is ported to OVS bridge: br-int

# ovs-vsctl show | grep 7008
        Port "tap70081ac2-6f"
            tag: 1
            Interface "tap70081ac2-6f"
                type: internal

• I didn't cover the HAProxy software because my point is only on tun/tap interface and Linux network space. But, how do I find the HAProxy process running on this network space?

# netns=qlbaas-13185f35-3f75-47e7-9fd7-301be7b28e88

# find -L /proc/[1-9]*/task/*/ns/net -samefile /run/netns/"$netns" | cut -d/ -f5

19937 <== the process id

# ps aux | grep 19937

root     14216  0.0  0.0  10432   932 pts/0    S+   02:29   0:00 grep --color=auto 19937

nobody   19937  0.0  0.0  29176  1472 ?        Ss   Mar16   0:06 haproxy -f /var/lib/neutron/lbaas/13185f35-3f75-47e7-9fd7-301be7b28e88/conf -p /var/lib/neutron/lbaas/13185f35-3f75-47e7-9fd7-301be7b28e88/pid -sf 8433

# ip netns identify 19937
qlbaas-13185f35-3f75-47e7-9fd7-301be7b28e88 <== where the namespace the process id is in

Get it. Here we go. So, put all the information together and then we can more understand how LBaaS implements.

[Fuel] How to use postgres database in Fuel

For those who wants to check out the data in Fuel's Postgres database, this document can give a simple guide to do so for the reference.
Here it is:

Find out the postgres docker

[root@fuel /]# docker ps


[root@fuel /]# dockerctl fuel-core-7.0-psotgres shell
[root@fuel /]# sudo su - postgres
-bash-4.1$ psql
psql (9.3.5)
Type "help" for help.

postgres=#

So, now we can use postgres sql database!

Use "nailgun" database

postgres=# \c nailgun

You are now connected to database "nailgun" as user "postgres".

List all tables in database

nailgun=# \dt

Look "tasks" table schema

nailgun=# \d tasks       

           

Use SQL to select the data in table

nailgun=# select * from information_schema.columns where table_name = tasks;

[Fuel] To build up Fuel Nailgun developing environment

  Well, I try to build up Fuel Nail Gun developing environment on my Debian 7 virtualbox VM. All my reference document is this nailgun.
  Although this document seems to use Ubuntu 12.04/14.04, but I still insist to build on Debian 7 due to my personal persistence. Here we go:

• Install nailgun developing environment

sudo apt-get install --yes postgresql postgresql-server-dev-all
sudo sed -ir 's/peer/trust/' /etc/postgresql/9.1/main/pg_hba.conf
sudo service postgresql restart
sudo -u postgres psql -c "CREATE ROLE nailgun WITH LOGIN PASSWORD 'nailgun'"
sudo -u postgres createdb nailgun

sudo apt-get install --yes python-dev python-pip

sudo pip install virtvualenv virtualewrapper
. /usr/local/bin/virtualenvwrapper.sh  # you can save this to .bashrc
mkvirtualenv fuel # you can use any name instead of 'fuel'
workon fuel  # command selects the particular environment

sudo apt-get install --yes git
git clone https://github.com/openstack/fuel-web.git
cd fuel-web
pip install --allow-all-external -r nailgun/test-requirements.txt

cd nailgun

python setup.py develop
sudo mkdir /var/log/nailgun
sudo chown -R `whoami`.`whoami` /var/log/nailgun

sudo chmod -R a+w /var/log/nailgun

sudo apt-get remove --yes nodejs nodejs-legacy
sudo apt-get install --yes software-properties-common
sudo add-apt-repository --yes ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install --yes nodejs

# This is only for Debian to do installing NPM
git clone https://github.com/joyent/node.git
cd node
#Now, if you require a specific version of Node:
git tag # Gives you a list of released versions
git checkout v0.4.12
# Then compile and install Node like this:
./configure
make
sudo make install

cd ~/fuel-web

sudo npm install -g gulp
sudo chown -R `whoami`.`whoami` ~/.npm
cd nailgun

# To install dependency packages for using Fuel UI

# Then compile and install Node like this:
npm install

• Start the nailgun environment

. /usr/local/bin/virtualenvwrapper.sh

workon fuel
cd fuel-web/nailgun
./manage.py syncdb
./manage.py loaddefault # It loads all basic fixtures listed in settings.yaml
./manage.py loaddata nailgun/fixtures/sample_environment.json  # Loads fake nodes

python manage.py run -p 8000 --fake-tasks

now, we can login Fuel UI by http://localhost:8000

• Install the Fuel8.0 client

Use Fuel API to program your installation tool 

https://pypi.python.org/pypi/python-fuelclient/8.0.0

• Other reference URLs:

Using Fuel CLI

https://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#using-fuel-cli

Fuel Plugin Catalog
https://www.mirantis.com/products/openstack-drivers-and-plugins/fuel-plugins/

Fuel Reference Architecture
https://docs.mirantis.com/openstack/fuel/fuel-7.0/reference-architecture.html

fuel-specs/specs/8.0/
https://github.com/openstack/fuel-specs/tree/master/specs/8.0

OpenStack/fuel-dev-tools
https://github.com/openstack/fuel-dev-tools

http://www.yet.org/2015/10/mos7-reducedfootprint/