I had a need for an application that I am writing on Google App Engine for a way to store jobs and then process them all at once. I found the idea for a Memcached lockless queue and created an implementation of it: queue.py
To write to it (I do this from a view where I want to store some stats about the data that was shown in the view):
thisQueue=queue.Queue(QUEUE_NAME) def method(): thisQueue.write(data)
and then later on to read from it. I created a cron job that is executed often.
thisQueue=queue.Queue(QUEUE_NAME) def cronMethod(): msg = thisQueue.read() while msg: processMessage(msg) msg = thisQueue.read()
If you are using this on Google App Engine, you should also be careful that you don’t run out of time to execute. If you get a DeadlineExceededError after you have read, but before you have finished processing, then the message might get lost.