redredred

I had a need for an application that I am writing on Google App Engine for a way to store jobs and then process them all at once. I found the idea for a Memcached lockless queue and created an implementation of it: queue.py
To write to it (I do this from a view where I want to store some stats about the data that was shown in the view):

thisQueue=queue.Queue(QUEUE_NAME)
def method():
    thisQueue.write(data)

and then later on to read from it. I created a cron job that is executed often.

thisQueue=queue.Queue(QUEUE_NAME)
def cronMethod():
    msg = thisQueue.read()
    while msg:
        processMessage(msg)
        msg = thisQueue.read()

If you are using this on Google App Engine, you should also be careful that you don’t run out of time to execute. If you get a DeadlineExceededError after you have read, but before you have finished processing, then the message might get lost.