Wednesday, October 7, 2015

[Python] How can I patch a Python decorator before it wraps a function?

As title, Python decorator could be a problem if you want to write some unit cases that test these functions which contains decorators because you cannot use Mock to patch your decorators. Why? Please refer to the following link to check out:

http://stackoverflow.com/questions/7667567/can-i-patch-a-python-decorator-before-it-wraps-a-function

Decorators are applied at function definition time. For most functions, this is when the module is loaded. (Functions that are defined in other functions have the decorator applied each time the enclosing function is called.)
So if you want to monkey-patch a decorator, what you need to do is:
  1. Import the module that contains it
  2. Define the mock decorator function
  3. Set e.g. module.decorator = mymockdecorator
  4. Import the module(s) that use the decorator, or use it in your own module

Let me give an exmple:
Here is a decorator function(user_passes_test) in the following file path
/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py

If you have a function using this decorator and need to write the test case for it as follows:

@user_passes_test(filter_root,
                  login_url=settings.LOGIN_PAGE)
def my_dashboard_view(request):


Then, you can put your patch code in the beginning of the tests python file like this:

def mock_function(test_func, login_url=None, redirect_field_name=None):
    def wrapped_f(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def decorator_view(request, *args, **kwargs):
            return view_func(request, *args, **kwargs)
        return decorator_function
    return wrapped_f

from django.contrib.auth.decorators import user_passes_test
user_passes_test = mock_function

After doing so, the decorator:  user_passes_test() won't infect your test case any more~

Thursday, October 1, 2015

[SSH] To examine the Linux PAM for login mechanism

Based on Linux PAM module, we can easily configure the related pam file and use the functionality that it provides. I just examine login authentication mechanism from PAM a little bit. The scenario is like this:
  I change the login authentication mechanism from key pair to password and to see what happens in /var/log/auth.log if I give the wrong password.


/etc/ssh/sshd_config

The value to change:
PubkeyAuthentication yes ==> no
PasswordAuthentication no ==> yes

$> service ssh restart

After restarting ssh daemon, I give the wrong password and then I can see the error message in the log file:

The following image is for reference if everything is correct.


Wednesday, September 2, 2015

[Make] Build the Linux Kernel

This document keeps my previous job that is to deal with re-building Linux Kernel for myself reference:

Modules存放位置:  /lib/modules/$(uname -r)/kernel/

核心檔案放置在:  /usr/src/kernels/`uname -r`/arch/x86/boot/bzImage

第一次進行編譯, 保持乾淨原始碼: make mrproper
其餘的時刻,你想要刪除前一次編譯過程的殘留資料: make clean

開始挑選核心功能: make XXconfig
make menuconfig

make help

[root@www linux-2.6.30.3]# make vmlinux  <==未經壓縮的核心
[root@www linux-2.6.30.3]# make modules  <==僅核心模組
[root@www linux-2.6.30.3]# make bzImage  <==經壓縮過的核心(預設)
[root@www linux-2.6.30.3]# make all      <==進行上述的三個動作


移動核心到 /boot 且保留舊核心檔案
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/arch/x86/boot/bzImage \
> /boot/vmlinuz-2.6.30.3vbird  <==實際核心
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/.config \
> /boot/config-2.6.30.3vbird   <==建議設定檔也複製備

建立相對應的 Initial Ram Disk (initrd)
mkinitrd -v /boot/initrd-2.6.30.3vbird.img  2.6.30.3vbird

編輯開機選單
vim /boot/grub/menu.lst

[Horizon] The summary of Horizon module

Basically Horizon is based on Django framework to be developed.

Install Horizon

To install the related packages for Horizon in Controller Node:
root@controller:~# apt-get -y install openstack-dashboard apache2 libapache2-mod-wsgi memcached python-memcache

Django setting 

Modify the file: /etc/openstack-dashboard/local_settings.py
Change the LOCATION's ip to controller's ip 
OPENSTACK_HOST = "controller"

# session ( default using memcached as session management )
CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': '192.168.22.5:11211',
   }
}

# Time zone
TIME_ZONE = "Asia/Taipei"

Restart the service

Restart apache & memcached:
root@controller:~# service apache2 restart
root@controller:~# service memcached restart

Reference

The following list and items are the the related information when I study Horizon.
Openstack introduction:
Trace python code:
OpenStack APIs and WSGI
SWGI

[Python] Protecting your Python code when doing the deployment

As we know that Python is a scripting language and need interpreter to compile the source code to byte code and execute it. When running python code, the interpreter will read Python file(*.py) and generate compiled Python file(*.pyc) during the first time running or doing the import. Unfortunately, there is no perfect way to protect your Python byte code because there are some tools, like: Python 2.7 decompiler (uncompyle2), can do the reverse engineering.

Here are several tools and approach that can do some kind of protection for your code.

1. Remove the Python source code
> python -m compileall .
or > python -OO -m compileall .
find . -name "*.py" -exec rm -rf {} \;

2. Let Obfuscater help you to obfuscate your code
Python Obfuscater
https://github.com/astrand/pyobfuscate/

3. If your python program is stand alone application, I believe PythonInstaller can help you generate an executable file for your Python code.
https://github.com/pyinstaller/pyinstaller/wiki

4. Use Cython
http://cython.org/
http://laing20333.blogspot.tw/

Thursday, August 27, 2015

[DPDK] OpenStack and DPDK

In my previous article "Install Lagopus software switch on Ubuntu 12.04", the virtual switch, Lagopus just heavily leverage DPDK to enhance the line rate near to the physical NIC capacity. Recently I saw an article "Scaling NFV to 213 Million Packets per Second with Red Hat Enterprise Linux, OpenStack, and DPDK", which mentions using DPDK in OpenStack and test its performance in RedHat Enterprise. Again DPDK is adopted in the network virtualization under OpenStack. Even thought there are some hardware specification limitation, for instance, with Intel x86 CPU and some series of network ethernet cards. But, I believe it will become one of standard option, just like SR-IOV, to deploy OpenStack in the future.

If you want to get started OpenStack with DPDK, I suggest to study Open vSwitch with DPDK first because the networking of virtual machines are all related with Open vSwitch.
https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md

Here are more related information for reference:

Accelerating Neutron with Intel DPDK
http://www.slideshare.net/AlexanderShalimov/ashalimov-neutron-dpdkv1

Intel DPDK Step by Step instructions
http://www.slideshare.net/hisaki/intel-dpdk-step-by-step-instructions

git: stackforge/networking-ovs-dpdk
http://git.openstack.org/cgit/stackforge/networking-ovs-dpdk

Friday, August 21, 2015

[Shell] ShellEd, the Shell Script Editor for Eclipse

If someone wants to use shell script editor in Eclipse, I think ShellEd is good candidate and its provides features, like syntax highlighting. The installation steps are here:

go Help => install new software => Add 
Name: ShellEd 
Location: http://www.chasetechnology.co.uk/eclipse/updates

Friday, August 7, 2015

[Docker] What is the difference between Docker and LXC?

As we know Docker is a hot topic in recent cloud conference or summit, for instance, OpenStack. Google also announced that they will donate its open source project "Kubernets" and integrate it into OpenStack. The news definitely cheers up a lot of OpenStackers. For me, I use LXC before, but don't know too much about Docker. So, I am curious the difference between Docker and LXC. Here it is:
https://www.flockport.com/lxc-vs-docker/
This content in the URL gives me the answer for my question.
http://stackoverflow.com/questions/17989306/what-does-docker-add-to-lxc-tools-the-userspace-lxc-tools
Docker is not a replacement for lxc. "lxc" refers to capabilities of the linux kernel (specifically namespaces and control groups) which allow sandboxing processes from one another, and controlling their resource allocations.
On top of this low-level foundation of kernel features, Docker offers a high-level tool with several powerful functionalities:
  • Portable deployment across machines. Docker defines a format for bundling an application and all its dependencies into a single object which can be transferred to any docker-enabled machine, and executed there with the guarantee that the execution environment exposed to the application will be the same. Lxc implements process sandboxing, which is an important pre-requisite for portable deployment, but that alone is not enough for portable deployment. If you sent me a copy of your application installed in a custom lxc configuration, it would almost certainly not run on my machine the way it does on yours, because it is tied to your machine's specific configuration: networking, storage, logging, distro, etc. Docker defines an abstraction for these machine-specific settings, so that the exact same docker container can run - unchanged - on many different machines, with many different configurations.
  • Application-centric. Docker is optimized for the deployment of applications, as opposed to machines. This is reflected in its API, user interface, design philosophy and documentation. By contrast, the lxc helper scripts focus on containers as lightweight machines - basically servers that boot faster and need less ram. We think there's more to containers than just that.
  • Automatic build. Docker includes a tool for developers to automatically assemble a container from their source code, with full control over application dependencies, build tools, packaging etc. They are free to use make, maven, chef, puppet, salt, debian packages, rpms, source tarballs, or any combination of the above, regardless of the configuration of the machines.
  • Versioning. Docker includes git-like capabilities for tracking successive versions of a container, inspecting the diff between versions, committing new versions, rolling back etc. The history also includes how a container was assembled and by whom, so you get full traceability from the production server all the way back to the upstream developer. Docker also implements incremental uploads and downloads, similar to "git pull", so new versions of a container can be transferred by only sending diffs.
  • Component re-use. Any container can be used as an "base image" to create more specialized components. This can be done manually or as part of an automated build. For example you can prepare the ideal python environment, and use it as a base for 10 different applications. Your ideal postgresql setup can be re-used for all your future projects. And so on.
  • Sharing. Docker has access to a public registry (https://registry.hub.docker.com/) where thousands of people have uploaded useful containers: anything from redis, couchdb, postgres to irc bouncers to rails app servers to hadoop to base images for various distros. The registry also includes an official "standard library" of useful containers maintained by the docker team. The registry itself is open-source, so anyone can deploy their own registry to store and transfer private containers, for internal server deployments for example.
  • Tool ecosystem. Docker defines an API for automating and customizing the creation and deployment of containers. There are a huge number of tools integrating with docker to extend its capabilities. PaaS-like deployment (Dokku, Deis, Flynn), multi-node orchestration (maestro, salt, mesos, openstack nova), management dashboards (docker-ui, openstack horizon, shipyard), configuration management (chef, puppet), continuous integration (jenkins, strider, travis), etc. Docker is rapidly establishing itself as the standard for container-based tooling.

Reference:
Docker in technical details ( Chinese Version)
http://www.cnblogs.com/feisky/p/4105739.html