Python matplotlib 中文亂碼解決 in Linux or Windows

 大家是否有遇過當使用matplotlib時，遇到顯示不出中文或是中文為亂碼的問題。無論是在Anaconda內使用matplotlib或是直接在Python安裝環境內使用matplotlib，基本解決方式是一樣的。本文將以簡單與條列方式描述在Linux 或是 Windows 解決方法。

1. 下載字型的來源

https://www.fontpalace.com/font-download/SimHei/

2. 字型安裝

執行python command, 並以下列方式找到matplotlib放置ttf字型檔位置與設定檔

>>> import matplotlib

>>> matplotlib.matplotlib_fname()

根據下列不同環境的執行結果內放置ttf字型檔

Windows: 

放到: C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\fonts\ttf 

修改設定檔  C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\matplotlibrc

Linux:

 

放到: /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf

修改設定檔  /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/matplotlibrc

Anaconda:

放到: /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf

修改設定檔  /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/matplotlibrc

3. 設定檔matplotlibrc修改方式

font.family  : sans-serif

font.serif      : SimHei   <== 增加此項目

範例如下:

4. 重新載入字體

>>> from matplotlib.font_manager import _rebuild

>>> _rebuild() 

USDT Python Tracing 範例

 Reference for USDT Tracepoint for Python

https://github.com/paulross/dtrace-py

https://www.collabora.com/news-and-blog/blog/2019/05/14/an-ebpf-overview-part-5-tracing-user-processes/

https://github.com/iovisor/bcc/pull/698

Install build tools and python prerequisites

sudo apt install systemtap-sdt-dev

sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev

Download and extract python

wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz

tar xf Python-3.7.0.tar.xz

cd Python-3.7.0

curl -o Python-3.7.0.tgz https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz

tar -xzf Python-3.7.0.tgz

cd Python-3.7.0

[Python] The issues of converting Python2 to Python3

If you are working on converting Python2 source code to Python3, there are some issues you will encounter sooner or later. I arrange my part here and will continue to update it as follows:

[Fun] compress and composite dataset to one image file

If you want to compress and composite the images in a dataset, like MNIST or CIFAR-10, you probably can refer to my example.

People need to see it to believe it. Here you go:

[Confusion Matrix] How to calculate confusion matrix, precision and recall list from scratch

I directly give an example which is with 10 categories, such as CIFAR-10 and MNIST. It explains how to calculate the confusion matrix, precision and recall list from scratch in Python. My data is generated at random. You should replace by yours. Here it goes:

[Caffe] Try out Caffe with Python code

This document is just a testing record to try out on Caffe with Python code. I refer to this blog. For using Python, we can easily to access every data flow blob in layers, including diff blob, weight blob and bias blob. It is so convenient for us to understand the change of training phase's weights and what have done in each step.

[Python] Problem with Python logging RotatingFileHandler in Django website

If seeing the log files are not rotated properly or correctly in Django web site, you most likely encounter the problem as the following article described:

Problem with Python logging RotatingFileHandler in Django website

"The log is done via RotatingFileHandler which is configured with 10 log files, 1000000 byte each. The log system works, but this are the log files I get:

-rw-r--r-- 1 apache      apache          83 Jul 23 13:30 hr.log
-rw-r--r-- 1 apache      apache      446276 Jul 23 13:03 hr.log.1
-rw-r--r-- 1 apache      apache      999910 Jul 23 06:00 hr.log.10
-rw-r--r-- 1 apache      apache         415 Jul 23 16:24 hr.log.2
-rw-r--r-- 1 apache      apache      479636 Jul 23 16:03 hr.log.3
-rw-r--r-- 1 apache      apache         710 Jul 23 15:30 hr.log.4
-rw-r--r-- 1 apache      apache      892179 Jul 23 15:03 hr.log.5
-rw-r--r-- 1 apache      apache         166 Jul 23 14:30 hr.log.6
-rw-r--r-- 1 apache      apache      890769 Jul 23 14:03 hr.log.7
-rw-r--r-- 1 apache      apache      999977 Jul 23 12:30 hr.log.8
-rw-r--r-- 1 apache      apache      999961 Jul 23 08:01 hr.log.9

As you can see it is a mess. Last log has been written to file hr.log.2 (Jul 23 16:24) instead of hr.log"

I did some survey and found the root cause is in here:
RotatingFileHandler bugs/errors and a general logging question
"The logging system is thread-safe but not safe
against multiple processes (separate Python instances) writing to the
same file. It certainly sounds like you need a scalable solution - and
having each script send the events to a network logging server seems a
good way of handling the scalability requirement. "

    These words above remind me a lot about the importance of synchronization when using multi-threading and multi-process. Also, scaling is another important item that a lot of people don't care. I want to highlight that we should be vigilant about these.

So, it is not your fault. Don't blame yourself. ( Sorry, I am kidding you ... )
Here is one solution for this issue. Please check out the following link ( it's written in Chinese ):
http://www.djangochina.cn/forum.php?mod=viewthread&tid=118752

[Python] How to get key and value from a custom header or from a form data with django rest framework

The following python code is part of handling a post method of my Restful API. I only excerpt it because I want to only hightlight my experiment of dealing with key and value from a custom HTTP header or from a form data.

""" Deal with the header property """
header_property_kwargs = {}
for field in request.META:
    logger.debug("...[Testing]...field,value==> %s,%s" % (field, request.META[field]))
    if 'HTTP_X_EXTRA_PROPERTY_' in field:        
     new_field = field.replace('HTTP_X_EXTRA_PROPERTY_','').lower()
        header_property_kwargs[new_field] = request.META[field]
logger.debug("...[Testing]...header_properties==> %s" % header_property_kwargs)

""" Deal with the extra_property  """
extra_property_kwargs = {}
if 'extra_property' in request.DATA.keys():
    extra_property = request.DATA['extra_property']
    try:
        property_list = extra_property.split("|")
        for property_item in property_list:
            key_value = property_item.split("=")
            extra_property_kwargs[key_value[0]] = key_value[1]
    except Exception as e:
        logger.error("Cannot parse extra_property" % e)
        return Response(status=status.HTTP_400_BAD_REQUEST)
logger.debug("...[Testing]...extra_properties==> %s" % extra_property_kwargs)

If I use the curl command to try my code, I will get the result:

curl -v -X POST -H 'Content-Type: application/json; indent=4' -H 'x-extra-property-version1: ccccc' -H 'x-extra-property-version2: ddddd' -u teyen.liu@gmail.com:123456 -d '{"extra_property":"version1=aaa|version2=bbb"}'  http://10.0.2.10:8000/api/test/

[Testing]...header_properties==> {'version1': 'ccccc', 'version2': 'ddddd'}

[Testing]...extra_properties==> {u'version1': u'aaa', u'version2': u'bbb'}

It proves that both approaches are able to do the same job. 

P.S: From HTTP Header it seams to use "X-" prefix string otherwise the parameter won't appear in request's header.


Reference:
http://stackoverflow.com/questions/28345842/getting-custom-header-on-post-request-with-django-rest-framework

curl --header "X-MyHeader: 123" --data "test=test" http://127.0.0.1:8000/api/update_log/

The name of the meta data attribute of request is in upper case:

print request.META

Your header will be available as:

request.META['HTTP_X_MYHEADER']

Or:

request.META.get('HTTP_X_MYHEADER') # return `None` if no such header

[Python] The a little thing about argument and kwargs

http://stackoverflow.com/questions/1769403/understanding-kwargs-in-python

You can use **kwargs to let your functions take an arbitrary number of keyword arguments:

>>> def print_keyword_args(**kwargs):
...     # kwargs is a dict of the keyword args passed to the function
...     for key, value in kwargs.iteritems():
...         print "%s = %s" % (key, value)
... 
>>> print_keyword_args(first_name="John", last_name="Doe")
first_name = John
last_name = Doe

You can also use the **kwargs syntax when calling functions by constructing a dictionary of keyword arguments and passing it to your function:

>>> kwargs = {'first_name': 'Bobby', 'last_name': 'Smith'}
>>> print_keyword_args(**kwargs)
first_name = Bobby
last_name = Smith

func(**{'type':'Event'})

is equivalent to

func(type='Event')

http://stackoverflow.com/questions/988228/converting-a-string-to-dictionary
How to convert a string to dict?

Starting in Python 2.6 you can use the built-in ast.literal_eval:

>>> import ast
>>> ast.literal_eval("{'muffin' : 'lolz', 'foo' : 'kitty'}")
{'muffin': 'lolz', 'foo': 'kitty'}

This is safer than using eval. As its own docs say:

>>> help(ast.literal_eval)
Help on function literal_eval in module ast:

literal_eval(node_or_string)
    Safely evaluate an expression node or a string containing a Python
    expression.  The string or node provided may only consist of the following
    Python literal structures: strings, numbers, tuples, lists, dicts, booleans,
    and None.

[Python] How can I patch a Python decorator before it wraps a function?

As title, Python decorator could be a problem if you want to write some unit cases that test these functions which contains decorators because you cannot use Mock to patch your decorators. Why? Please refer to the following link to check out:

http://stackoverflow.com/questions/7667567/can-i-patch-a-python-decorator-before-it-wraps-a-function

Decorators are applied at function definition time. For most functions, this is when the module is loaded. (Functions that are defined in other functions have the decorator applied each time the enclosing function is called.)

So if you want to monkey-patch a decorator, what you need to do is:

1. Import the module that contains it
2. Define the mock decorator function
3. Set e.g. module.decorator = mymockdecorator
4. Import the module(s) that use the decorator, or use it in your own module

Let me give an exmple:

Here is a decorator function(user_passes_test) in the following file path

/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py

If you have a function using this decorator and need to write the test case for it as follows:

@user_passes_test(filter_root,
                  login_url=settings.LOGIN_PAGE)
def my_dashboard_view(request):


Then, you can put your patch code in the beginning of the tests python file like this:


def mock_function(test_func, login_url=None, redirect_field_name=None):
    def wrapped_f(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def decorator_view(request, *args, **kwargs):
            return view_func(request, *args, **kwargs)
        return decorator_function
    return wrapped_f

from django.contrib.auth.decorators import user_passes_test
user_passes_test = mock_function


After doing so, the decorator:  user_passes_test() won't infect your test case any more~

[Python] Protecting your Python code when doing the deployment

As we know that Python is a scripting language and need interpreter to compile the source code to byte code and execute it. When running python code, the interpreter will read Python file(*.py) and generate compiled Python file(*.pyc) during the first time running or doing the import. Unfortunately, there is no perfect way to protect your Python byte code because there are some tools, like: Python 2.7 decompiler (uncompyle2), can do the reverse engineering.

Here are several tools and approach that can do some kind of protection for your code.

1. Remove the Python source code
> python -m compileall .
or > python -OO -m compileall .
find . -name "*.py" -exec rm -rf {} \;

2. Let Obfuscater help you to obfuscate your code
Python Obfuscater
https://github.com/astrand/pyobfuscate/

3. If your python program is stand alone application, I believe PythonInstaller can help you generate an executable file for your Python code.
https://github.com/pyinstaller/pyinstaller/wiki

4. Use Cython
http://cython.org/
http://laing20333.blogspot.tw/

[Python] Effective Python

Two weeks ago, my friend discussed the best practice for Django and Python in programming.
After that for a while, one day I just came up with a idea: I read a book "Effective C++" and why not there is a book about "Effective Python"? So, here it is: http://www.effectivepython.com/
I wish I can find some time to walk through this book.

[Python] A simple example of Python Egg

The following content is based on this article:
http://www.mxm.dk/2008/02/python-eggs-simple-introduction.html

hello/
  _wrapper.so
  hello.py ==>
    def helloworld():
        print 'Hello World'


1. First, prepare your setup.py ==>
from distutils.core import setup # Distutils

setup(name='hello',
    version='1.0',
    packages=['hello',],
    package_data={
        'hello': ['*.so'],
    }
)

2. Run python setup.py sdist

3. Change a line in your setup.py ==>
#from distutils.core import setup # Distutils
from setuptools import setup, find_packages # Egg

setup(name='hello',
    version='1.0',
    packages=['hello',],
    package_data={
        'hello': ['*.so'],
    }
)

4. Run  python setup.py bdist_egg

5. Then you get a new file in your dist directory:
dist/
hello-1.0-py2.4.egg

My example files and output package are here:


6. To install the egg file
> sudo easy_install hello-1.0-py2.7.egg
in case if there is an error for easy_install
> sudo apt-get install python-setuptools

[Python] Make pyc file for your Python source code

It could be a situation when you have to give your Python program to customers, but you don't want to give them Python source code. Here is a solution. Give them Python byte code!

For example, on command line:

• Compile one file

         > python -c "import compileall; compileall.compile_file('YourPythonFile.py')"

• Compile one folder

         > python -c "import compileall; compileall.compile_dir('YourFolder/', force=True)"

Or, you can put the compiling action in source code

      import compileall
      compileall.compile_dir('YourFolder/', force=True)

[Python] How to use Decorators for Functions and Methods

When list_ports() is executed, Class ApiCall will be created for becoming a decorator to examine the arguments as follows:

import sys class ApiCall(object): """A Decorator to add support for format and tenant overriding""" def __init__(self, function): self.function = function def __get__(self, instance, owner): def with_params(*args, **kwargs): """ Temporarily sets the format and tenant for this request """ (format, tenant) = (instance.format, instance.tenant) if 'format' in kwargs: instance.format = kwargs['format'] if 'tenant' in kwargs: instance.tenant = kwargs['tenant'] ret = self.function(instance, *args) (instance.format, instance.tenant) = (format, tenant) return ret return with_params class Client(object): def __init__(self, tenant=None, format="xml"): self.tenant = tenant self.format = format @ApiCall def list_ports(self, network): """ Fetches a list of ports on a given network """ return network def main(): client = Client(tenant="AAA",format="xml") client.list_ports('my network') sys.exit(0) if __name__ == "__main__": main()