Memcache lockless queue implementation

Jonathan Ricketson

I had a need for an application that I am writing on Google App Engine for a way to store jobs and then process them all at once. I found the idea for a Memcached lockless queue and created an implementation of it:
To write to it (I do this from a view where I want to store some stats about the data that was shown in the view):

def method():

and then later on to read from it. I created a cron job that is executed often.

def cronMethod():
    msg =
    while msg:
        msg =

If you are using this on Google App Engine, you should also be careful that you don’t run out of time to execute. If you get a DeadlineExceededError after you have read, but before you have finished processing, then the message might get lost.

This entry was posted on June 24th, 2009.