Really simple python message queue

5 Comments

  • Sharebar

Sometime we were looking for python message queue.  Our requirements were pretty simple.

Problem:
Our problem was to update the recent commit data into a database table through Subversion’s post-commit hook.  Since this is a post-commit hook, if it takes a long time, user may get ‘timeout errors’. That is not acceptable. Since sometimes commits are quite large,  we have to put the the task in queue and process it later. One option is to start a a separate process to update the table. However, we are using django. Hence starting a new process mean ‘restarting django’ which will be a significant overhead.

So we wanted

  1. Something which keeps queue in memory or in database. Persistent queue was not needed.
  2. implemented in python. Since It was very simple requirement. Hence we didn’t want the overhead of setting up another application and maintaining it.
  3. Today there will be a single django process extracting tasks from queue and executing them. However, tomorrow this process may be executed on another server.
  4. simple queue client (preferably without any direct database dependency)

We looked at various options.

  1. Celery/django-celery
  2. ActiveMQ + python client
  3. PyMQ
  4. Few more.

However, all these approach seemed to be a over kill for our requirements. I was thinking about this requirement for 2/3 days. Then suddenly a thought struck me.  Email server maintains a queue of email message and python has a simple SMTP class.  I can use this class to implement a message queue.

  1. Derive a class from smtpd.SMTPServer.
  2. Override ‘process_message’ method.
  3. In ‘process_message’ start a thread.
  4. Inside the thread function, read the message contents and execute the task.
  5. The message contents are simple JSON objects.
  6. Client code is simple. Client just have to send a ‘email’ to this local SMTP server. Send the task parameters encoded in JSON format as content of this email. So the client can be a simple shell script.

UPDATE : It took me 1 hour to code this. Its a single python file of about 40 lines. As the title says its ‘really simple message queue’. There are no configurations to maintain.  Since this particular Python SMTP server is configured to listen to a non-standard port and receive mails only from local machine,  ’injecting’ nasty messages is not serious issue.
In future, when we need some serious queue server, we will use something else.

5 Responses to “Really simple python message queue”

  1. Failguy

    This is truly an epic fail. Use a simple queue such as beanstalkd. I love the:

    “I was thinking about this requirement for 2/3 days.”

    And then you hack up an email based solution which took who knows how long. I hope your solution accounts for failed delivery, and message injection: “The message contents are simple JSON objects.”

    So instead of maintaining a separate application (the queue server like beanstalkd) you now have to maintain the code for this solution, and he has to maintain an email server and whatever accounts go with this (and that email server better we well protected since it sounds like you could trivially inject nasty messages). This is a fail on so many levels.

    Reply
    • nitinbhide

      It took me 1 hour to code this. Its a single python file of about 40 lines. Obviously it was not great effort. I don’t think it is going to result in huge efforts in maintaining this single file. As the title says its ‘really simple message queue’.

      There are no configurations to maintain. I am not running standard SMTP server so there is no accounts to configure. Since this particular SMTP server is configured to listen to a non-standard port and receive mails only from local machine, ‘injecting’ nasty messages is not serious issue.

      In future, when we need some serious queue server, we will use something else.

      Reply
  2. NIH much?

    So I agree with failguy, looks like NIH syndrome, you’ve taken an issue which has many, many well known and well described solutions, and implemented it as a custom standalone queue based around sending and receiving emails.

    Sounds fun and easy to maintain. I’m sure finding a developer 4 years down the line who can also maintain this will be easy…

    I also like that you spent 2-3 days thinking about this! Perhaps if you had spent 20 minutes exploring the well known solutions, you wouldn’t have reimplemented this one!

    Reply
    • nitinbhide

      I don’t expect it to survive for 4 years. We will probably go for different solution in a year or two (may be less). We know it is a short term solution.

      We have spent much more than 20 minutes exploring other solutions. All well known solutions require learning, configuring and regularly maintaining new product technology etc. For really small team, it is an overhead.

      Reply

Leave a Reply