[GitHub] How to rebase your branch from upstream branch?

If you fork a copy (branch) from a repository and you want to rebase your branch with this upstream repository, you can refer to the following URLs or directly see the content below.

https://help.github.com/articles/configuring-a-remote-for-a-fork/
https://help.github.com/articles/syncing-a-fork/

1. List the current configured remote repository for your fork.

   git remote -v
   # origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
   # origin  https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
   

2. Specify a new remote upstream repository that will be synced with the fork.

   git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git
   

3. Verify the new upstream repository you've specified for your fork.

   git remote -v
   # origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
   # origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
   # upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (fetch)
   # upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (push)
   

1. Fetch the branches and their respective commits from the upstream repository. Commits tomaster will be stored in a local branch, upstream/master.

   git fetch upstream
   # remote: Counting objects: 75, done.
   # remote: Compressing objects: 100% (53/53), done.
   # remote: Total 62 (delta 27), reused 44 (delta 9)
   # Unpacking objects: 100% (62/62), done.
   # From https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY
   #  * [new branch]      master     -> upstream/master
   

2. Check out your fork's local master branch.

   git checkout master
   # Switched to branch 'master'
   

3. Merge the changes from upstream/master into your local master branch. This brings your fork's master branch into sync with the upstream repository, without losing your local changes.

   git merge upstream/master
   # Updating a422352..5fdff0f
   # Fast-forward
   #  README                    |    9 -------
   #  README.md                 |    7 ++++++
   #  2 files changed, 7 insertions(+), 9 deletions(-)
   #  delete mode 100644 README
   #  create mode 100644 README.md
   

   If your local branch didn't have any unique commits, Git will instead perform a "fast-forward":

   git merge upstream/master
   # Updating 34e91da..16c56ad
   # Fast-forward
   #  README.md                 |    5 +++--
   #  1 file changed, 3 insertions(+), 2 deletions(-)

[Django] How to Find the Performance Bottlenecks in Your Django Views?

Do you want to find the performance bottlenecks in your Django Views? I recently found this article that introduces nice tools for us to check the performance issues in Django. How to do it? Please follow this URL:
http://djangotricks.blogspot.tw/2015/01/performance-bottlenecks-in-django-views.html

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
INSTALLED_APPS = (
    # ...
    'devserver',
)

MIDDLEWARE_CLASSES = (
    # ...
    'devserver.middleware.DevServerMiddleware',
)

DEVSERVER_MODULES = (
    'devserver.modules.sql.SQLRealTimeModule',
    'devserver.modules.sql.SQLSummaryModule',
    'devserver.modules.profile.ProfileSummaryModule',

    # Modules not enabled by default
    'devserver.modules.profile.LineProfilerModule',
)

DEVSERVER_AUTO_PROFILE = True  # profiles all views without the need of function decorator

I try these tools and get a bunch of analysis report from them, and I think they are useful and convenient. The following picture is one part of my result:

[Selenium] How to use xpath to define the locator in Selenium Library?

When I first time touch Selenium library in Robot Framework, the first question mark coming up in my mind is "How to use xpath to define the locator?"

 Here is an example. If I want to select the listbox that is near by my email td tag, and the HTML looks like the follows:

There could be several ways to do so. But, I like to use this way to achieve the action that I described below. "Select From List" is the standard keyword to do the action for selecting a list, but you have to give the locator. Based on my HTML code, the answer is this:

Another example, if I want to select the checkbox that is near by my email td tag, I can use the following way to do so:

Or, if I want to get some information in a table, what can I do for this?
OK, if I have the table's HTML code like this:

I can use "Get Table Cell" keyword with xpath //table[@class='your table's class name'] and the information in which row and column, then you can finish the task to get the data from the cell in the table. Here you go:


Reference:
http://stackoverflow.com/questions/247135/using-xpath-to-search-text-containing

[Python] How can I patch a Python decorator before it wraps a function?

As title, Python decorator could be a problem if you want to write some unit cases that test these functions which contains decorators because you cannot use Mock to patch your decorators. Why? Please refer to the following link to check out:

http://stackoverflow.com/questions/7667567/can-i-patch-a-python-decorator-before-it-wraps-a-function

Decorators are applied at function definition time. For most functions, this is when the module is loaded. (Functions that are defined in other functions have the decorator applied each time the enclosing function is called.)

So if you want to monkey-patch a decorator, what you need to do is:

1. Import the module that contains it
2. Define the mock decorator function
3. Set e.g. module.decorator = mymockdecorator
4. Import the module(s) that use the decorator, or use it in your own module

Let me give an exmple:

Here is a decorator function(user_passes_test) in the following file path

/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py

If you have a function using this decorator and need to write the test case for it as follows:

@user_passes_test(filter_root,
                  login_url=settings.LOGIN_PAGE)
def my_dashboard_view(request):


Then, you can put your patch code in the beginning of the tests python file like this:


def mock_function(test_func, login_url=None, redirect_field_name=None):
    def wrapped_f(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def decorator_view(request, *args, **kwargs):
            return view_func(request, *args, **kwargs)
        return decorator_function
    return wrapped_f

from django.contrib.auth.decorators import user_passes_test
user_passes_test = mock_function


After doing so, the decorator:  user_passes_test() won't infect your test case any more~

[SSH] To examine the Linux PAM for login mechanism

Based on Linux PAM module, we can easily configure the related pam file and use the functionality that it provides. I just examine login authentication mechanism from PAM a little bit. The scenario is like this:
  I change the login authentication mechanism from key pair to password and to see what happens in /var/log/auth.log if I give the wrong password.

/etc/ssh/sshd_config

The value to change:

PubkeyAuthentication yes ==> no

PasswordAuthentication no ==> yes

$> service ssh restart

After restarting ssh daemon, I give the wrong password and then I can see the error message in the log file:

The following image is for reference if everything is correct.

[Make] Build the Linux Kernel

This document keeps my previous job that is to deal with re-building Linux Kernel for myself reference:

Modules存放位置:  /lib/modules/$(uname -r)/kernel/

核心檔案放置在:  /usr/src/kernels/`uname -r`/arch/x86/boot/bzImage

第一次進行編譯, 保持乾淨原始碼： make mrproper
其餘的時刻，你想要刪除前一次編譯過程的殘留資料: make clean

開始挑選核心功能： make XXconfig
make menuconfig

make help

[root@www linux-2.6.30.3]# make vmlinux  <==未經壓縮的核心
[root@www linux-2.6.30.3]# make modules  <==僅核心模組
[root@www linux-2.6.30.3]# make bzImage  <==經壓縮過的核心(預設)
[root@www linux-2.6.30.3]# make all      <==進行上述的三個動作


移動核心到 /boot 且保留舊核心檔案
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/arch/x86/boot/bzImage \
> /boot/vmlinuz-2.6.30.3vbird  <==實際核心
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/.config \
> /boot/config-2.6.30.3vbird   <==建議設定檔也複製備

建立相對應的 Initial Ram Disk (initrd)
mkinitrd -v /boot/initrd-2.6.30.3vbird.img  2.6.30.3vbird

編輯開機選單
vim /boot/grub/menu.lst

[Horizon] The summary of Horizon module

Basically Horizon is based on Django framework to be developed.

Install Horizon

To install the related packages for Horizon in Controller Node:

root@controller:~# apt-get -y install openstack-dashboard apache2 libapache2-mod-wsgi memcached python-memcache

Django setting 

Modify the file: /etc/openstack-dashboard/local_settings.py
Change the LOCATION's ip to controller's ip 

OPENSTACK_HOST = "controller"

# session ( default using memcached as session management )
CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': '192.168.22.5:11211',
   }
}

# Time zone
TIME_ZONE = "Asia/Taipei"

Restart the service

Restart apache & memcached：

root@controller:~# service apache2 restart
root@controller:~# service memcached restart

Reference

The following list and items are the the related information when I study Horizon.



Openstack introduction:

http://www.slideshare.net/ya790026/openstack-34821569

Trace python code:

https://github.com/ya790206/call_seq

OpenStack APIs and WSGI

http://www.slideshare.net/lhrc_mikeyp/openstack-apis-and-wsgi?next_slideshow=1

SWGI

http://www.slideshare.net/mingli.yuan/a-brief-introduce-to-wsgi?next_slideshow=2

Pyreverse : UML Diagrams for Python (Logilab.org)

[Python] Protecting your Python code when doing the deployment

As we know that Python is a scripting language and need interpreter to compile the source code to byte code and execute it. When running python code, the interpreter will read Python file(*.py) and generate compiled Python file(*.pyc) during the first time running or doing the import. Unfortunately, there is no perfect way to protect your Python byte code because there are some tools, like: Python 2.7 decompiler (uncompyle2), can do the reverse engineering.

Here are several tools and approach that can do some kind of protection for your code.

1. Remove the Python source code
> python -m compileall .
or > python -OO -m compileall .
find . -name "*.py" -exec rm -rf {} \;

2. Let Obfuscater help you to obfuscate your code
Python Obfuscater
https://github.com/astrand/pyobfuscate/

3. If your python program is stand alone application, I believe PythonInstaller can help you generate an executable file for your Python code.
https://github.com/pyinstaller/pyinstaller/wiki

4. Use Cython
http://cython.org/
http://laing20333.blogspot.tw/

[DPDK] OpenStack and DPDK

In my previous article "Install Lagopus software switch on Ubuntu 12.04", the virtual switch, Lagopus just heavily leverage DPDK to enhance the line rate near to the physical NIC capacity. Recently I saw an article "Scaling NFV to 213 Million Packets per Second with Red Hat Enterprise Linux, OpenStack, and DPDK", which mentions using DPDK in OpenStack and test its performance in RedHat Enterprise. Again DPDK is adopted in the network virtualization under OpenStack. Even thought there are some hardware specification limitation, for instance, with Intel x86 CPU and some series of network ethernet cards. But, I believe it will become one of standard option, just like SR-IOV, to deploy OpenStack in the future.

If you want to get started OpenStack with DPDK, I suggest to study Open vSwitch with DPDK first because the networking of virtual machines are all related with Open vSwitch.
https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md

Here are more related information for reference:

Accelerating Neutron with Intel DPDK
http://www.slideshare.net/AlexanderShalimov/ashalimov-neutron-dpdkv1

Intel DPDK Step by Step instructions
http://www.slideshare.net/hisaki/intel-dpdk-step-by-step-instructions

git: stackforge/networking-ovs-dpdk
http://git.openstack.org/cgit/stackforge/networking-ovs-dpdk

[Shell] ShellEd, the Shell Script Editor for Eclipse

If someone wants to use shell script editor in Eclipse, I think ShellEd is good candidate and its provides features, like syntax highlighting. The installation steps are here:

go Help => install new software => Add 
Name: ShellEd 
Location: http://www.chasetechnology.co.uk/eclipse/updates




[Docker] What is the difference between Docker and LXC?

As we know Docker is a hot topic in recent cloud conference or summit, for instance, OpenStack. Google also announced that they will donate its open source project "Kubernets" and integrate it into OpenStack. The news definitely cheers up a lot of OpenStackers. For me, I use LXC before, but don't know too much about Docker. So, I am curious the difference between Docker and LXC. Here it is:
https://www.flockport.com/lxc-vs-docker/

This content in the URL gives me the answer for my question.
http://stackoverflow.com/questions/17989306/what-does-docker-add-to-lxc-tools-the-userspace-lxc-tools

Docker is not a replacement for lxc. "lxc" refers to capabilities of the linux kernel (specifically namespaces and control groups) which allow sandboxing processes from one another, and controlling their resource allocations.

On top of this low-level foundation of kernel features, Docker offers a high-level tool with several powerful functionalities:

• Portable deployment across machines. Docker defines a format for bundling an application and all its dependencies into a single object which can be transferred to any docker-enabled machine, and executed there with the guarantee that the execution environment exposed to the application will be the same. Lxc implements process sandboxing, which is an important pre-requisite for portable deployment, but that alone is not enough for portable deployment. If you sent me a copy of your application installed in a custom lxc configuration, it would almost certainly not run on my machine the way it does on yours, because it is tied to your machine's specific configuration: networking, storage, logging, distro, etc. Docker defines an abstraction for these machine-specific settings, so that the exact same docker container can run - unchanged - on many different machines, with many different configurations.
• Application-centric. Docker is optimized for the deployment of applications, as opposed to machines. This is reflected in its API, user interface, design philosophy and documentation. By contrast, the lxc helper scripts focus on containers as lightweight machines - basically servers that boot faster and need less ram. We think there's more to containers than just that.
• Automatic build. Docker includes a tool for developers to automatically assemble a container from their source code, with full control over application dependencies, build tools, packaging etc. They are free to use make, maven, chef, puppet, salt, debian packages, rpms, source tarballs, or any combination of the above, regardless of the configuration of the machines.
• Versioning. Docker includes git-like capabilities for tracking successive versions of a container, inspecting the diff between versions, committing new versions, rolling back etc. The history also includes how a container was assembled and by whom, so you get full traceability from the production server all the way back to the upstream developer. Docker also implements incremental uploads and downloads, similar to "git pull", so new versions of a container can be transferred by only sending diffs.
• Component re-use. Any container can be used as an "base image" to create more specialized components. This can be done manually or as part of an automated build. For example you can prepare the ideal python environment, and use it as a base for 10 different applications. Your ideal postgresql setup can be re-used for all your future projects. And so on.
• Sharing. Docker has access to a public registry (https://registry.hub.docker.com/) where thousands of people have uploaded useful containers: anything from redis, couchdb, postgres to irc bouncers to rails app servers to hadoop to base images for various distros. The registry also includes an official "standard library" of useful containers maintained by the docker team. The registry itself is open-source, so anyone can deploy their own registry to store and transfer private containers, for internal server deployments for example.
• Tool ecosystem. Docker defines an API for automating and customizing the creation and deployment of containers. There are a huge number of tools integrating with docker to extend its capabilities. PaaS-like deployment (Dokku, Deis, Flynn), multi-node orchestration (maestro, salt, mesos, openstack nova), management dashboards (docker-ui, openstack horizon, shipyard), configuration management (chef, puppet), continuous integration (jenkins, strider, travis), etc. Docker is rapidly establishing itself as the standard for container-based tooling.

Reference:
Docker in technical details ( Chinese Version)
http://www.cnblogs.com/feisky/p/4105739.html

[Raspberry Pi] What job do I use my Pi to do?

I bought my Raspberry Pi last year and didn't use it often. The main reason is that I didn't figure out what job I use my Pi to do. Until several months ago, I finally made the decision: My Pi will become a torrent server...Nice.

There is a very good advantage to use Pi to act as torrent server: "low power consumption"
So, here is the steps to do:
http://www.takobear.tw/2014/12/07/rpiraspberry-pibt-2/
http://www.takobear.tw/2014/03/20/rpiraspberry-pibt/

P.S: This web site also has a lot of tutorials for Raspberry Pi, check it out: http://www.takobear.tw/category/tutorial/rpi/

[Linux] “No such file or directory” on files that exist?

First, this kind of problem is almost related with dynamic loader. You can check out what it is:
To understand Linux Dynamic Loader
http://www.cs.virginia.edu/~dww4s/articles/ld_linux.html

After verifying it with several ways:
# file "your binary"
# readelf -l  "your binary"
# strings "your binary" | grep ld-

Then you can find out which dynamic loader your binary uses.
For my case, I just create a symbolic link to my loader, and then it works.

liudanny@Debian7 x64_lsb $ ll /lib64/
total 8
drwxr-xr-x  2 root root 4096 Aug  2 20:49 .
drwxr-xr-x 24 root root 4096 Jun 20 18:14 ..
lrwxrwxrwx  1 root root   32 Feb 22 06:41 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.13.so
lrwxrwxrwx  1 root root   20 Aug  2 20:49 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2

I also find some other issue about running a 32-bit binary on a 64-bit system. Here is a good answer in the following link:
Quote:
"But the program is a 32-bit program (as the file output indicates), looking for the 32-bit loader /lib/ld-linux.so.2, and you've presumably only installed the 64-bit loader /lib64/ld-linux-x86-64.so.2 in the chroot."
http://unix.stackexchange.com/questions/13391/getting-not-found-message-when-running-a-32-bit-binary-on-a-64-bit-system



To install ia32-libs on debian wheezy amd64:

dpkg --add-architecture i386 
apt-get update
apt-get install libc6:i386

or:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo aptitude install ia32-libs

[Python] Effective Python

Two weeks ago, my friend discussed the best practice for Django and Python in programming.
After that for a while, one day I just came up with a idea: I read a book "Effective C++" and why not there is a book about "Effective Python"? So, here it is: http://www.effectivepython.com/
I wish I can find some time to walk through this book.

[Neutron] Slow network speed between VM and external

Recently I encountered a situation of the VM's network performance that is pretty low. For instance, if upload a image to Glance via my VM to run glance cli command, the actual transmit speed is around the following data:

RX: 130 Kbps

TX: 150 Kbps

My OpenStack's network service is with GRE segmentation on Neutron. Why? After google the web, I found some people had the same issues as I had:

https://ask.openstack.org/en/question/25306/slow-network-speed-between-vm-and-external/

They talked about the approach on the VM to solve the issue and this works

> ethtool -K eth0 gro off

> ethtool -K eth0 tso off

But, for my case I think it is not solve from the root cause so that I take time to do the research for TCP segmentation offload and GRE. Finally I find the root cause:
MTU need to adjust when using GRE network service.
http://docs.openstack.org/juno/install-guide/install/yum/content/neutron-network-node.html
Quote:
"Tunneling protocols such as GRE include additional packet headers that increase overhead and decrease space available for the payload or user data. Without knowledge of the virtual network infrastructure, instances attempt to send packets using the default Ethernet maximum transmission unit (MTU) of 1500 bytes.Internet protocol (IP) networks contain the path MTU discovery (PMTUD) mechanism to detect end-to-end MTU and adjust packet size accordingly. However, some operating systems and networks block or otherwise lack support for PMTUD causing performance degradation or connectivity failure.
Ideally, you can prevent these problems by enabling jumbo frames on the physical network that contains your tenant virtual networks. Jumbo frames support MTUs up to approximately 9000 bytes which negates the impact of GRE overhead on virtual networks. However, many network devices lack support for jumbo frames and OpenStack administrators often lack control over network infrastructure. Given the latter complications, you can also prevent MTU problems by reducing the instance MTU to account for GRE overhead. Determining the proper MTU value often takes experimentation, but 1454 bytes works in most environments. You can configure the DHCP server that assigns IP addresses to your instances to also adjust the MTU."

After doing the following setting on Controller node, the transmit speed is up to:

RX: 31323 Kbps

TX: 31464 Kbps

1. Edit the /etc/neutron/dhcp_agent.ini file and complete the following action:
   1. In the [DEFAULT] section, enable the dnsmasq configuration file:
      

      1 2 3

      [DEFAULT] ... dnsmasq_config_file = /etc/neutron/dnsmasq-neutron.conf

2. Create and edit the /etc/neutron/dnsmasq-neutron.conf file and complete the following action:
   1. Enable the DHCP MTU option (26) and configure it to 1454 bytes:
      

      1	dhcp-option-force=26,1454

3. Kill any existing dnsmasq processes:

   # pkill dnsmasq

So, the network performance is improved 200 times.
Wow...



Reference:
TCP segmentation offload
https://forum.ivorde.com/linux-tso-tcp-segmentation-offload-what-it-means-and-how-to-enable-disable-it-t19721.html
TCP in Linux Kernel
http://vger.kernel.org/~davem/tcp_output.html
Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environmenthttps://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140

[OF-DPA] A simple L2 switch RYU app for OF-DPA

I found there is a official OF-DPA web site that contains some good information about RYU and OF-DPA as follows:
http://ofdpa.com/

I also put my simple L2 switch RYU app for OF-DPA which I had done an example in the last year and cannot guarantee there is no error/bug. But, it is still worth referencing. Enjoy.
Here it is:
https://github.com/teyenliu/OF-DPA