Tuesday, May 3, 2016

[Python] Problem with Python logging RotatingFileHandler in Django website

If seeing the log files are not rotated properly or correctly in Django web site, you most likely encounter the problem as the following article described:

Problem with Python logging RotatingFileHandler in Django website
"The log is done via RotatingFileHandler which is configured with 10 log files, 1000000 byte each. The log system works, but this are the log files I get:
-rw-r--r-- 1 apache      apache          83 Jul 23 13:30 hr.log
-rw-r--r-- 1 apache      apache      446276 Jul 23 13:03 hr.log.1
-rw-r--r-- 1 apache      apache      999910 Jul 23 06:00 hr.log.10
-rw-r--r-- 1 apache      apache         415 Jul 23 16:24 hr.log.2
-rw-r--r-- 1 apache      apache      479636 Jul 23 16:03 hr.log.3
-rw-r--r-- 1 apache      apache         710 Jul 23 15:30 hr.log.4
-rw-r--r-- 1 apache      apache      892179 Jul 23 15:03 hr.log.5
-rw-r--r-- 1 apache      apache         166 Jul 23 14:30 hr.log.6
-rw-r--r-- 1 apache      apache      890769 Jul 23 14:03 hr.log.7
-rw-r--r-- 1 apache      apache      999977 Jul 23 12:30 hr.log.8
-rw-r--r-- 1 apache      apache      999961 Jul 23 08:01 hr.log.9
As you can see it is a mess. Last log has been written to file hr.log.2 (Jul 23 16:24) instead of hr.log"

I did some survey and found the root cause is in here:
RotatingFileHandler bugs/errors and a general logging question
"The logging system is thread-safe but not safe
against multiple processes (separate Python instances) writing to the
same file. It certainly sounds like you need a scalable solution - and
having each script send the events to a network logging server seems a
good way of handling the scalability requirement. "

    These words above remind me a lot about the importance of synchronization when using multi-threading and multi-process. Also, scaling is another important item that a lot of people don't care. I want to highlight that we should be vigilant about these.

So, it is not your fault. Don't blame yourself. ( Sorry, I am kidding you ... )
Here is one solution for this issue. Please check out the following link ( it's written in Chinese ):
http://www.djangochina.cn/forum.php?mod=viewthread&tid=118752

No comments: