Yesterday almost all BootStrapToday users received, Ticket modification notifications for all tickets assigned to them. (probably lot of emails notifications). First our apologies to our users for annoying them with so many mails. We have learned our lesson and we have taken the corrective steps to ensure that this does not happen again.
Short Version of what happened:
Yesterday we upgraded BootStrapToday to new version. As part of this upgrade, we added a new field ‘Complexity’ to the tickets. For all existing ‘closed’ tickets , complexity is set to ‘Unkown’, other tickets ‘complexity’ is set to ‘Simple’. Since this is a change in the ticket, our ticket notification system automatically triggered notifications to all these tickets. Somehow we missed this when we upgraded our QA server for testing. By the time we realized what has happened, the system has sent notifications to almost all users.
Long Version of what happened (to technically interested):
The BootStrapToday uses Django framework. Our upgrade process is automated using ‘Fabric”. We use celery daemon to run background processing tasks and trigger email notifications.
The moment ticket is saved , a background task is added the celery task. This celery task determines to whom notifications should go.
The upgrade process roughly goes through following steps (these are Fabric steps and hence automated).
- stop all services. (including celery)
- Update the source.
- Execute schema changes (e.g. django syncdb)
- Execute scripts to update any existing data (as required).
For example, like adding a default complexity to Tickets. In our case, we did not do this while adding ‘complexity’ column to tickets table. We added separate post ‘schema changes’ script. We queried at all ‘closed’ tickets, set the complexity to ‘Unknown’ and saved the ticket. Each ticket save added a celery task for notifications. - Restart all services (e.g. web server, celery daemon etc).
At this point, all there were large number of celery tasks for ticket notification and the moment celery daemon started, it started executing the these tasks and started sending emails.
Since upgrade is an automated process, by time we realized what is happening and stopped the celery daemon, hundreds of email notifications were already sent.
What are we doing to ensure that it does not happen again ?
We have added a flag to disable email notifications in our ‘data update’ step. If the emails are to be completely disabled, then we intercept the Django EmailMessage.send() function and stop the email from sent. There is also an additional check on disabling some notification types instead of all. If we need to notify users of some specific data update, we will put in separate ‘inform user’ step.
Once again our apologies to our users for annoying them with so many mails. We have learned our lesson and we have taken the corrective steps to ensure that this does not happen again.
PS: Given below is quick simple code on intercepting EmailMessasge.send()
from django.core.mail import EmailMessage def intercept_email_send(email_send_func): def mail_send(*args, **kwargs): #check if email is enabled in settings. If not, don't send the mail. if settings.ENABLE_EMAIL: email_send_func(*args, **kwargs) return mail_send EmailMessage.send = intercept_email_send(EmailMessage.send)