This article is a follow-up to our previous tutorial on how to create a custom Log Handler for sending specific logs as Slack messages.
Let's say you have implemented a [.c-inline-code]SlackHandler[.c-inline-code], and at some point in your code you end up performing an HTTP request against the slack API. Usually, this kind of operation is I/O blocking, so the execution thread will wait for this call to finish to continue its own execution, i.e. your code will be blocked until it gets a response from Slack.
The drawback here is that if something goes wrong with the external service such as the request hanging for a while, it may trigger a timeout exception on your server as the original request is taking as much time or more than your request against that external service.
This means that any of your processes could potentially be killed due to these kinds of errors. Since this shutdown mechanism may occur on higher layers of your architecture, i.e. Nginx, Apache, etc, you might not notice this behavior fast enough.
So, a way to handle this is to separate the logging thread from the main process to ensure any error or delay during logging will not affect the execution of your code. As an example, I'll highlight how to avoid this in a clean manner, using Python's [.c-inline-code]QueueHandler[.c-inline-code] and [.c-inline-code]QueueListener[.c-inline-code] classes.
QueueHandler
According to Python’s documentation, this handler is “located in the logging.handlers module, and supports sending logging messages to a queue, such as those implemented in the queue or multiprocessing modules."
We'll use this handler to "collect" any log messages we want to send to Slack and then store them in a queue. To achieve this, the first modification we are going to do is in the settings file of your Django project. See here how to do so.
This change means that instead of using our [.c-inline-code]SlackHandler[.c-inline-code] directly, we want to enqueue the messages on [.c-inline-code]SLACK_QUEUE[.c-inline-code]. These are all the modifications we need regarding the [.c-inline-code]QueueHandler[.c-inline-code]. Next, we need a way to grab those enqueued messages and send them to our [.c-inline-code]SlackHandler[.c-inline-code] so they actually reach Slack, we need to create and start a [.c-inline-code]QueueListener[.c-inline-code].
QueueListener
Again, according to Python: "The QueueListener class, located in the logging.handlers module supports receiving logging messages from a queue, such as those implemented in the queue or multiprocessing modules. The messages are received from a queue in an internal thread and passed, on the same thread, to one or more handlers for processing."
To do this, we need to instantiate a [.c-inline-code]QueueListener[.c-inline-code], configuring it with the queue we want to use, and the handlers we want to redirect the messages to.
For this example, I chose to put this startup code in two places. I defined a method to make the actual startup, and then I put a call on [.c-inline-code]my_app/apps.py[.c-inline-code] inside the ready method, to ensure the listener initialization occurs just once in the application's execution.
So, in the [.c-inline-code]slack.py[.c-inline-code] file, you will need to follow these steps. Then, on [.c-inline-code]my_app/apps.py[.c-inline-code], you will need to follow this method.
And, that's it, your log messages will now go directly into the queue we defined before, and then, on a separate thread, will be processed by [.c-inline-code]QueueListener[.c-inline-code], which will put them in the hands of our [.c-inline-code]SlackHandler[.c-inline-code].
You can test this by intentionally putting a [.c-inline-code]sleep()[.c-inline-code] call on [.c-inline-code]SlackHandler[.c-inline-code] and see how your code will continue its execution. After the sleep ends, you will see the messages coming to slack.
Summary
In this breakdown, we described how to use a separate thread to handle heavy-load log operations using a queue, how to use a [.c-inline-code]QueueHandler[.c-inline-code] to put items into the queue, and how to use a [.c-inline-code]QueueListener[.c-inline-code] to get items from that queue to send into other handlers, i.e.the [.c-inline-code]SlackHandler[.c-inline-code].
