100

I am playing with Python's logging system. I have noticed a strange behavior while removing handlers from a Logger object in a loop. Namely, my for loop removes all but one handler. Additional call to .removeHandler removes the last handler smoothly. No error messages are issued during the calls.

This is the test code:

import logging
import sys
logging.basicConfig()
dbg = logging.getLogger('dbg')
dbg.setLevel(logging.DEBUG)

testLogger = logging.getLogger('mylogger')
sh = logging.StreamHandler(sys.stdout)
fh = logging.FileHandler('mylogfile.log')
dbg.debug('before adding handlers: %d handlers'%len(testLogger.handlers))
testLogger.addHandler(fh)
testLogger.addHandler(sh)

dbg.debug('before removing. %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
for h in testLogger.handlers:
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

#HERE I EXPECT THAT NO HANDLER WILL REMAIN    
dbg.debug('after removing: %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
if len(testLogger.handlers) > 0:
    #Why is this happening?
    testLogger.removeHandler(testLogger.handlers[0])
dbg.debug('after manually removing the last handler: %d handlers'%len(testLogger.handlers))

I expect that at the end of the loop no handlers will remain in the testLogger object, however the last call to .removeHandler apparently fails, as can be seen from the output below. Nevertheless additional call to this function removes the handler as expected. Here is the output:

DEBUG:dbg:before adding handlers: 0 handlers
DEBUG:dbg:before removing. 2 handlers: [<logging.FileHandler instance at 0x021263F0>, <logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:removing handler <logging.FileHandler instance at 0x021263F0>
DEBUG:dbg:1 more to go
DEBUG:dbg:after removing: 1 handlers: [<logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:after manually removing the last handler: 0 handlers

More interestingly, if I replace the original loop with the following one, the loop works as expected and no handlers remain in the testLogger object at the end of the loop. Here is the modified loop:

while len(testLogger.handlers) > 0:
    h = testLogger.handlers[0]
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

What explains this behaviour? Is this a bug or am I missing something?

Improve this question
1
  • 10
    for h in list(testLogger.handlers) Commented Jul 19, 2014 at 0:04

5 Answers 5

Reset to default
173

This isn't logger-specific behaviour. Never mutate (insert/remove elements) the list you're currently iterating on. If you need, make a copy. In this case testLogger.handlers.clear() should do the trick.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Solution is not so obvious and nice since we need to take into account thread safety which is obviously compromised here (take a look at addHandler implementation - locks are used there to modify handlers list).
A safer alternative would be to modify the existing list: testLogger.handlers.clear()
@zenofpython I suppose testLogger.handlers.clear() calls the clear method of list type? Why is it safter than testLogger.handlers = []?
Redefining a list rather than modifying the original list is a problem whenever you use an alias somewhere that points to the original list. Suppose: a = b = [1, 2, 3] a = [] Notice that b still points to [1, 2, 3]. In fairness, it is unlikely that you would ever use such an alias in this case, but it is good practice to maintain consistency so you don't get any unexpected surprises from cutting corners.
Note this is not proper solution (when one considers thread-safety). According to the Python docs on logging.Logger.handlers, .handlers attribute should be considered a read-only.
33

instead of mutating undocumented .handler:

Option 1

logging.getLogger().removeHandler(logging.getLogger().handlers[0])

this way you remove exactly the preexisting handler object via offical api. Or to remove all handlers:

logger = logging.getLogger()
while logger.hasHandlers():
    logger.removeHandler(logger.handlers[0])

Option 2

logging.config.dictConfig(config={'level': logging.DEBUG, 'handlers': []})

Not only removes but prevents its creation. List root will have [] handlers

Improve this answer

3 Comments

Option 1 does not work for non-root loggers: the method hasHandlers() also checks the handlers of the ancestor loggers, so if an ancestor logger has an handler then logger.hasHandlers() will always evaluate to True and therefore logger.handlers[0] will end up raising an IndexError.
Going off option 2, you can also just set your instantiated logger object's handlers attribute to an empty array, such as logger = logging.getLogger(__name__) and logger.handlers = []
@deesolie yes but safer to use .clear - see @zenofpython's comment
18

If you don't want to delete them all (thanks for the tip @CatPlusPlus):

testLogger.handlers = [
    h for h in testLogger.handlers if not isinstance(h, logging.StreamHandler)]
Improve this answer

5 Comments

Sorry,I found this didn't work.for isinstance can't used to judge handler's type
@fat_cheng Why can't isinstance work for that purpose? I don't seem to have a problem using the suggestion in this answer myself.
I'm using python3 in Ubuntu 16.04. I use if (type(h) == logging.StreamHandler)] instead.
@fat_cheng you should never do that because type doesn't check parent types (inheritance), isinstance does: stackoverflow.com/a/1549854/623735
For personnal purposes i wanted to check that a handker worked without spamming slack channel afterwards. handlers = logger.handlers; logger.error('Removing slack handler'); logger.handlers = [handler for handler in logger.handlers if handler != slack_handler]
18

I would say if the intent for removing the logging handlers is to prevent sending logs to additional handlers (this is the case most of the times) then an alternative approach I follow is to set propagate to False for your current logger.

logger = logging.getLogger(__name__)
logger.propagate = False

That way your log messages will only go to the handlers you explicitly add and will not propagate to parent loggers.

Improve this answer

Comments

3

I've just found out that you can also do that within a logging .ini file, with the following block:

[logger_stpipe]
handlers=
propagate=1
qualname=stpipe

It basically deactivates all handlers for a given logger. But it's somewhat limited because you have to know the Logger's name in advance.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.