[Django] How to Find the Performance Bottlenecks in Your Django Views?

Do you want to find the performance bottlenecks in your Django Views? I recently found this article that introduces nice tools for us to check the performance issues in Django. How to do it? Please follow this URL:
http://djangotricks.blogspot.tw/2015/01/performance-bottlenecks-in-django-views.html

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
INSTALLED_APPS = (
    # ...
    'devserver',
)

MIDDLEWARE_CLASSES = (
    # ...
    'devserver.middleware.DevServerMiddleware',
)

DEVSERVER_MODULES = (
    'devserver.modules.sql.SQLRealTimeModule',
    'devserver.modules.sql.SQLSummaryModule',
    'devserver.modules.profile.ProfileSummaryModule',

    # Modules not enabled by default
    'devserver.modules.profile.LineProfilerModule',
)

DEVSERVER_AUTO_PROFILE = True  # profiles all views without the need of function decorator

I try these tools and get a bunch of analysis report from them, and I think they are useful and convenient. The following picture is one part of my result:

[Selenium] How to use xpath to define the locator in Selenium Library?

When I first time touch Selenium library in Robot Framework, the first question mark coming up in my mind is "How to use xpath to define the locator?"

 Here is an example. If I want to select the listbox that is near by my email td tag, and the HTML looks like the follows:

There could be several ways to do so. But, I like to use this way to achieve the action that I described below. "Select From List" is the standard keyword to do the action for selecting a list, but you have to give the locator. Based on my HTML code, the answer is this:

Another example, if I want to select the checkbox that is near by my email td tag, I can use the following way to do so:

Or, if I want to get some information in a table, what can I do for this?
OK, if I have the table's HTML code like this:

I can use "Get Table Cell" keyword with xpath //table[@class='your table's class name'] and the information in which row and column, then you can finish the task to get the data from the cell in the table. Here you go:


Reference:
http://stackoverflow.com/questions/247135/using-xpath-to-search-text-containing

[Python] How can I patch a Python decorator before it wraps a function?

As title, Python decorator could be a problem if you want to write some unit cases that test these functions which contains decorators because you cannot use Mock to patch your decorators. Why? Please refer to the following link to check out:

http://stackoverflow.com/questions/7667567/can-i-patch-a-python-decorator-before-it-wraps-a-function

Decorators are applied at function definition time. For most functions, this is when the module is loaded. (Functions that are defined in other functions have the decorator applied each time the enclosing function is called.)

So if you want to monkey-patch a decorator, what you need to do is:

1. Import the module that contains it
2. Define the mock decorator function
3. Set e.g. module.decorator = mymockdecorator
4. Import the module(s) that use the decorator, or use it in your own module

Let me give an exmple:

Here is a decorator function(user_passes_test) in the following file path

/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py

If you have a function using this decorator and need to write the test case for it as follows:

@user_passes_test(filter_root,
                  login_url=settings.LOGIN_PAGE)
def my_dashboard_view(request):


Then, you can put your patch code in the beginning of the tests python file like this:


def mock_function(test_func, login_url=None, redirect_field_name=None):
    def wrapped_f(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def decorator_view(request, *args, **kwargs):
            return view_func(request, *args, **kwargs)
        return decorator_function
    return wrapped_f

from django.contrib.auth.decorators import user_passes_test
user_passes_test = mock_function


After doing so, the decorator:  user_passes_test() won't infect your test case any more~

[SSH] To examine the Linux PAM for login mechanism

Based on Linux PAM module, we can easily configure the related pam file and use the functionality that it provides. I just examine login authentication mechanism from PAM a little bit. The scenario is like this:
  I change the login authentication mechanism from key pair to password and to see what happens in /var/log/auth.log if I give the wrong password.

/etc/ssh/sshd_config

The value to change:

PubkeyAuthentication yes ==> no

PasswordAuthentication no ==> yes

$> service ssh restart

After restarting ssh daemon, I give the wrong password and then I can see the error message in the log file:

The following image is for reference if everything is correct.

[Make] Build the Linux Kernel

This document keeps my previous job that is to deal with re-building Linux Kernel for myself reference:

Modules存放位置:  /lib/modules/$(uname -r)/kernel/

核心檔案放置在:  /usr/src/kernels/`uname -r`/arch/x86/boot/bzImage

第一次進行編譯, 保持乾淨原始碼： make mrproper
其餘的時刻，你想要刪除前一次編譯過程的殘留資料: make clean

開始挑選核心功能： make XXconfig
make menuconfig

make help

[root@www linux-2.6.30.3]# make vmlinux  <==未經壓縮的核心
[root@www linux-2.6.30.3]# make modules  <==僅核心模組
[root@www linux-2.6.30.3]# make bzImage  <==經壓縮過的核心(預設)
[root@www linux-2.6.30.3]# make all      <==進行上述的三個動作


移動核心到 /boot 且保留舊核心檔案
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/arch/x86/boot/bzImage \
> /boot/vmlinuz-2.6.30.3vbird  <==實際核心
[root@www ~]# cp /usr/src/kernels/linux-2.6.30.3/.config \
> /boot/config-2.6.30.3vbird   <==建議設定檔也複製備

建立相對應的 Initial Ram Disk (initrd)
mkinitrd -v /boot/initrd-2.6.30.3vbird.img  2.6.30.3vbird

編輯開機選單
vim /boot/grub/menu.lst

[Horizon] The summary of Horizon module

Basically Horizon is based on Django framework to be developed.

Install Horizon

To install the related packages for Horizon in Controller Node:

root@controller:~# apt-get -y install openstack-dashboard apache2 libapache2-mod-wsgi memcached python-memcache

Django setting 

Modify the file: /etc/openstack-dashboard/local_settings.py
Change the LOCATION's ip to controller's ip 

OPENSTACK_HOST = "controller"

# session ( default using memcached as session management )
CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': '192.168.22.5:11211',
   }
}

# Time zone
TIME_ZONE = "Asia/Taipei"

Restart the service

Restart apache & memcached：

root@controller:~# service apache2 restart
root@controller:~# service memcached restart

Reference

The following list and items are the the related information when I study Horizon.



Openstack introduction:

http://www.slideshare.net/ya790026/openstack-34821569

Trace python code:

https://github.com/ya790206/call_seq

OpenStack APIs and WSGI

http://www.slideshare.net/lhrc_mikeyp/openstack-apis-and-wsgi?next_slideshow=1

SWGI

http://www.slideshare.net/mingli.yuan/a-brief-introduce-to-wsgi?next_slideshow=2

Pyreverse : UML Diagrams for Python (Logilab.org)

[Python] Protecting your Python code when doing the deployment

As we know that Python is a scripting language and need interpreter to compile the source code to byte code and execute it. When running python code, the interpreter will read Python file(*.py) and generate compiled Python file(*.pyc) during the first time running or doing the import. Unfortunately, there is no perfect way to protect your Python byte code because there are some tools, like: Python 2.7 decompiler (uncompyle2), can do the reverse engineering.

Here are several tools and approach that can do some kind of protection for your code.

1. Remove the Python source code
> python -m compileall .
or > python -OO -m compileall .
find . -name "*.py" -exec rm -rf {} \;

2. Let Obfuscater help you to obfuscate your code
Python Obfuscater
https://github.com/astrand/pyobfuscate/

3. If your python program is stand alone application, I believe PythonInstaller can help you generate an executable file for your Python code.
https://github.com/pyinstaller/pyinstaller/wiki

4. Use Cython
http://cython.org/
http://laing20333.blogspot.tw/

[DPDK] OpenStack and DPDK

In my previous article "Install Lagopus software switch on Ubuntu 12.04", the virtual switch, Lagopus just heavily leverage DPDK to enhance the line rate near to the physical NIC capacity. Recently I saw an article "Scaling NFV to 213 Million Packets per Second with Red Hat Enterprise Linux, OpenStack, and DPDK", which mentions using DPDK in OpenStack and test its performance in RedHat Enterprise. Again DPDK is adopted in the network virtualization under OpenStack. Even thought there are some hardware specification limitation, for instance, with Intel x86 CPU and some series of network ethernet cards. But, I believe it will become one of standard option, just like SR-IOV, to deploy OpenStack in the future.

If you want to get started OpenStack with DPDK, I suggest to study Open vSwitch with DPDK first because the networking of virtual machines are all related with Open vSwitch.
https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md

Here are more related information for reference:

Accelerating Neutron with Intel DPDK
http://www.slideshare.net/AlexanderShalimov/ashalimov-neutron-dpdkv1

Intel DPDK Step by Step instructions
http://www.slideshare.net/hisaki/intel-dpdk-step-by-step-instructions

git: stackforge/networking-ovs-dpdk
http://git.openstack.org/cgit/stackforge/networking-ovs-dpdk