Sometime we were looking for python message queue. Our requirements were pretty simple.
Our problem was to update the recent commit data into a database table through Subversion’s post-commit hook. Since this is a post-commit hook, if it takes a long time, user may get ‘timeout errors’. That is not acceptable. Since sometimes commits are quite large, we have to put the the task in queue and process it later. One option is to start a a separate process to update the table. However, we are using django. Hence starting a new process mean ‘restarting django’ which will be a significant overhead.
So we wanted
- Something which keeps queue in memory or in database. Persistent queue was not needed.
- implemented in python. Since It was very simple requirement. Hence we didn’t want the overhead of setting up another application and maintaining it.
- Today there will be a single django process extracting tasks from queue and executing them. However, tomorrow this process may be executed on another server.
- simple queue client (preferably without any direct database dependency)
We looked at various options.
However, all these approach seemed to be a over kill for our requirements. I was thinking about this requirement for 2/3 days. Then suddenly a thought struck me. Email server maintains a queue of email message and python has a simple SMTP class. I can use this class to implement a message queue.
- Derive a class from smtpd.SMTPServer.
- Override ‘process_message’ method.
- In ‘process_message’ start a thread.
- Inside the thread function, read the message contents and execute the task.
- The message contents are simple JSON objects.
- Client code is simple. Client just have to send a ‘email’ to this local SMTP server. Send the task parameters encoded in JSON format as content of this email. So the client can be a simple shell script.
UPDATE : It took me 1 hour to code this. Its a single python file of about 40 lines. As the title says its ‘really simple message queue’. There are no configurations to maintain. Since this particular Python SMTP server is configured to listen to a non-standard port and receive mails only from local machine, ’injecting’ nasty messages is not serious issue.
In future, when we need some serious queue server, we will use something else.