Feb 19 2010

Evolvethefuture is live

Jonathan

Finally EvolveTheFuture is live. There is still plenty to do, but now it has a reasonably stable API, and is ready for other people with enough documentation for people other than me to get started.

I am going to work next on community building, because it is going to be a lot more interesting if there are more people involved.

A excerpt from the EvolveTheFuture homepage:

EvolveTheFuture is an artificial life (Alife) environment that challenges you to write a better animal. It is a simple challenge with infinitely complex and interesting solutions.

Animals can move about eating other animals and are in turn eaten. Animals can reproduce and the children are usually like the parent, but sometimes they are a little different. And in that little difference evolution is allowed.

Evolution happens slowly though, so more interesting things might be able to be created through the imagination of people interested in natural competition. Animals are written in a custom assembly code and run in a javascript environment in your browser – and everyone else’s who opens this page.

The best animals are those that create a more successful species.

The best creators are those that create a successful dynasty.

Please note that this site works best in Firefox 3.5 and above. Chrome 3 and above and Safari 4 and above. It may not work at all in other browsers.

So, please come along and see what is happening. If you have any friends that you think might be interested, tell them too. And please tell me if you have any ideas or problems.

no comments

Feb 8 2010

Memcache lockless queue implementation (v2)

Jonathan

The new Google App Engine 1.3.1 SDK has an added method on memcache called “grab_tail”. This is a great little method that removes the need for my lockless queue to manage the read counter. We still need to manage the write counter though, because there doesn’t seem to be a way to add items to a memcache namespace with an arbitrary name.

I have also added a way to rollback in the case that your processing of queue items encounters an error. I was doing the reads non-destructively earlier, so I just needed to reset the read-counter to an appropriate point, but now (because grab_tail is destructive), we need to re-queue the messages.

<rant>

In this implementation, the method that gets the next write counter “__nextCounter”, is far more complicated than it should be, because of an outstanding defect on memcache. This defect is probably quite simple to fix, but since I reported it almost 6 months ago, it hasn’t even been acknowledged. I do really wonder about the value of an issue tracker to a community that doesn’t use it. If the google team are not going to use the issue tracker, it would be far better to acknowledge that and get rid of it completely.

</rant>

Here it is

from google.appengine.api import memcache
import logging
 
class Queue(object):
    itemPrefix="queueItem"
    writeCounter="writeCounter"
    def __init__(self, queueName):
        self.name = queueName
 
    def write(self, msg):
        counter = self.__nextWriteCounter()[0]
        msgKey = self.itemPrefix + str(counter)
        if not memcache.add(msgKey, msg, namespace=self.name):
            raise QueueException("msg key already existed: %s" % msgKey)
        logging.debug("wrote to %s:%s" % (self.name, msgKey))
 
    def writeMulti(self, messages):
        if len(messages) == 0:
            return
        mapping={}
        counters = self.__nextWriteCounter(len(messages))
        for msg,counter in zip(messages,counters):
            mapping[self.itemPrefix + str(counter)]=msg
 
        self.writeMulti(memcache.add_multi(mapping, namespace=self.name))
 
    def read(self):
        result = memcache.grab_tail(item_count=1, namespace=self.name)
        if len(result) > 0:
            return result[0]
        else:
            return None
 
    def readMulti(self, maxItems=100):
        return memcache.grab_tail(item_count=maxItems, namespace=self.name)
 
    def requeueMessages(self, messages):
        self.writeMulti(messages)
 
    def __currentWriteCounter(self):
        return self.__currentCounter(self.writeCounter, 0)
    def __nextWriteCounter(self,howmany=1):
        return self.__nextCounter(self.writeCounter, howmany=howmany)
 
    def __currentCounter(self, key, default):
        counter = memcache.get(key, namespace=self.name)
        if not counter:
            memcache.set(key, default, namespace=self.name)
            counter = default
        return counter
 
    def __nextCounter(self, key, howmany):
        counter = memcache.incr(key, namespace=self.name+"c", delta=howmany)
        if counter is None:
            if not memcache.add(key, howmany, namespace=self.name+"c"):
                # handles the case where another thread got in first with the add
                counter = memcache.incr(key, namespace=self.name+"c",delta=howmany)
                if counter==None:
                    raise QueueException("could not increment counter: %s" % key)
            else:
                counter = howmany
        return range(counter-(howmany-1),counter+1)
 
class QueueException(Exception):
    """Base APIProxy error type."""

and to use it:

queue=queue.Queue(clientViews.QUEUE_NAME)
def processMessages(request):
    msgs=[]
    try:
        msgs = clicksAndViewsQueue.readMulti(maxItems=100)
        if len(msgs) > 0:
            for msg in msgs:
                processMessage(msg)
 
            taskqueue.addTask(url=viewUrl, queueName=taskqueue.BACKGROUND_QUEUE)
    except Exception, e:
        clicksAndViewsQueue.requeueMessages(msgs)
        raise

no comments | posted in Development, google-app-engine, python