Problem with Python logging RotatingFileHandler in Django website
"The log is done via RotatingFileHandler which is configured with 10 log files, 1000000 byte each. The log system works, but this are the log files I get:
-rw-r--r-- 1 apache apache 83 Jul 23 13:30 hr.log
-rw-r--r-- 1 apache apache 446276 Jul 23 13:03 hr.log.1
-rw-r--r-- 1 apache apache 999910 Jul 23 06:00 hr.log.10
-rw-r--r-- 1 apache apache 415 Jul 23 16:24 hr.log.2
-rw-r--r-- 1 apache apache 479636 Jul 23 16:03 hr.log.3
-rw-r--r-- 1 apache apache 710 Jul 23 15:30 hr.log.4
-rw-r--r-- 1 apache apache 892179 Jul 23 15:03 hr.log.5
-rw-r--r-- 1 apache apache 166 Jul 23 14:30 hr.log.6
-rw-r--r-- 1 apache apache 890769 Jul 23 14:03 hr.log.7
-rw-r--r-- 1 apache apache 999977 Jul 23 12:30 hr.log.8
-rw-r--r-- 1 apache apache 999961 Jul 23 08:01 hr.log.9
As you can see it is a mess. Last log has been written to file hr.log.2 (Jul 23 16:24) instead of hr.log"
I did some survey and found the root cause is in here:
RotatingFileHandler bugs/errors and a general logging question
"The logging system is thread-safe but not safe
against multiple processes (separate Python instances) writing to the
same file. It certainly sounds like you need a scalable solution - and
having each script send the events to a network logging server seems a
good way of handling the scalability requirement. "
These words above remind me a lot about the importance of synchronization when using multi-threading and multi-process. Also, scaling is another important item that a lot of people don't care. I want to highlight that we should be vigilant about these.
So, it is not your fault. Don't blame yourself. ( Sorry, I am kidding you ... )
Here is one solution for this issue. Please check out the following link ( it's written in Chinese ):
http://www.djangochina.cn/forum.php?mod=viewthread&tid=118752
