redredred

Finally EvolveTheFuture is live. There is still plenty to do, but now it has a reasonably stable API, and is ready for other people with enough documentation for people other than me to get started.

I am going to work next on community building, because it is going to be a lot more interesting if there are more people involved.

A excerpt from the EvolveTheFuture homepage:

EvolveTheFuture is an artificial life (Alife) environment that challenges you to write a better animal. It is a simple challenge with infinitely complex and interesting solutions.

Animals can move about eating other animals and are in turn eaten. Animals can reproduce and the children are usually like the parent, but sometimes they are a little different. And in that little difference evolution is allowed.

Evolution happens slowly though, so more interesting things might be able to be created through the imagination of people interested in natural competition. Animals are written in a custom assembly code and run in a javascript environment in your browser – and everyone else’s who opens this page.

The best animals are those that create a more successful species.

The best creators are those that create a successful dynasty.

Please note that this site works best in Firefox 3.5 and above. Chrome 3 and above and Safari 4 and above. It may not work at all in other browsers.

So, please come along and see what is happening. If you have any friends that you think might be interested, tell them too. And please tell me if you have any ideas or problems.

After talking about the previous version of my migration script, I had need to make some significant changes to it. These changes support loading model classes that do not validate in their current model version. I.E. if you have added a new required field, or have renamed a field then you can make these changes using this script.

This uses the underlying Query and Entity types, so no Model constraints are enforced. But after you make all the changes to the entity, it is then loaded into the Model class to get any defaults applied and validation.

So the main features of this migration approach is:

• Uses the task queue for paging over all specified Model classes
• Sets defaults from the Model definition
• Allows adding/removing modifying of fields through a Dictionary like object (the Entity)

If you just want to apply any new defaults, you don’t need to do anything, other than specify the model class in the list to migrate. Any classes without a MigrationWorker are loaded into their Model class and have defaults and validation applied through that mechanism.

If there are specific things that you want to do to the object then you need to create your own MigrationWorker.

class ads_fooWorker(MigrationWorker):
    kind = "ads_foo"
    def processItem(self, item):
        logging.info("processing %s %s" % (item.kind(), item.key()))
        logging.info(item)
        if "views" in item and type(item['views']) == type([]):
            item['views'] = sum(item['views'])
        elif "views" not in item:
            item['views'] = 0
        if "clicks" in item and type(item['clicks']) == type([]):
            item['clicks'] = sum(item['clicks'])
        elif "clicks" not in item:
            item['clicks'] = 0
        super(ads_fooWorker, self).processItem(item)

In this class we are migrating a model class that (because we are using AEP) is called ads_foo. The underlying Entity object that we are operating on is a Dictionary like object, so we can look for keys, add keys and ‘del’ keys. As you can see here, we are changing a field from being a List of Integers to being a single Integer. This would not be possible if the object were loaded into the Model class.

The thing to be careful of (if using the taskQueue) is to make sure that you check if you have already made the changes as you are not guaranteed that your task will not be run twice. So make sure that your MigrationWorker checks if it needs to do it’s work.

The full code is here. To do my migration I usually add my MigrationWorkers directly to this script.

import logging
 
from django.http import HttpResponse
 
from google.appengine.api import datastore
from google.appengine.ext import db
from google.appengine.api.labs import taskqueue
from google.appengine.runtime import apiproxy_errors
 
"""migrate from version x to version y"""
def migrate(request):
    modelsToMigrate = ["ads_foo",
                     "ads_bar"]
    [_addTask(url=Worker.worker_url % i) for i in modelsToMigrate]
    return HttpResponse("created all tasks for processing")
 
#General Migration
def migrateModel(request, model_name):
    #create the worker class and tell it to work
    workerName = '%sWorker' % model_name
    if not workerName in globals():
        logging.info("no worker for %s" % model_name)
        Worker = generateWorker(model_name)
    else:
        Worker = globals()[workerName]
    if "start" in request.REQUEST:
        Worker(request.REQUEST['start']).work()
    else:
        Worker().work()
    return HttpResponse("ok")
 
class MigrationWorker(object):
    ITEMS_TO_FETCH = 10
    worker_url = "/worker/migrate/%s"
    def __init__(self, startKey=None):
        self.startKey = startKey
 
    def work(self):
        query = datastore.Query(self.kind)
        if self.startKey:
            query['__key__ >'] = db.Key(self.startKey)
        items = query.Get(self.ITEMS_TO_FETCH)
        if not items:
            logging.info('Finished migrating %s' % self.kind)
            return
 
        last_key = items[-1].key()
        [self.processItem(x) for x in items]
 
        _addTask(url=self.worker_url % self.kind, params=dict(start=last_key))
        logging.info('Added another task to queue for %s starting at %s' %
                     (self.kind, last_key))
 
    """Override this method to do some work for each item
    """
    def processItem(self, item):
        logging.info("processing %s %s" % (item.kind(), item.key()))
        modelClass = db.class_for_kind(item.kind()).from_entity(item)
        modelClass.put()
 
def generateWorker(kind_name):
    class DynamicClass(MigrationWorker):
        kind = kind_name
    return DynamicClass
 
class ads_fooWorker(MigrationWorker):
    kind = "ads_foo"
    def processItem(self, item):
        logging.info("processing %s %s" % (item.kind(), item.key()))
        logging.info(item)
        if "views" in item and type(item['views']) == type([]):
            item['views'] = sum(item['views'])
        elif "views" not in item:
            item['views'] = 0
        if "clicks" in item and type(item['clicks']) == type([]):
            item['clicks'] = sum(item['clicks'])
        elif "clicks" not in item:
            item['clicks'] = 0
        super(ads_fooWorker, self).processItem(item)
 
def _addTask(url, params={}, queueName='default'):
    try:
        logging.info("add task to %s [%s]" % (queueName, (url, params)))
        task = taskqueue.Task(url=url, params=params)
        task.add(queueName)
    except taskqueue.TransientError, e:
        logging.exception("adding Task failed with a TransientError")
        addTask(url, params, queueName)
    except apiproxy_errors.OverQuotaError, e:
        #but keep going
        logging.exception("adding Task failed with a TransientError")

Over at 37signals, Matt blogged on a topic recently that really resonated with me… the gist of what he said being that if a project or company is made up of a whole lot of people who don’t really know each other, individuals are generally going to play it safer than a group of people who are comfortable with one another who might fight harder to get their point heard. It doesn’t need to be a shit-fight, just an environment where people can be freely passionate and walk away as friends.  Obviously this is a generalisation and there are always people who will say what they feel – I tend to be one of them although that’s somewhat mood dependent. Anyway, the net effect of this can often be mediocrity which can be damaging or at least limiting to a project or company.

Something else that I find i’m often up against is trying to consider how important someone’s job is to them when I set expectations around quality or general awareness of what’s going on in their professional world. As someone who spends silly amounts of time working on code, reading about development / design etc at all hours of the day, I need to keep reminding myself that for many others it’s just a job and they’re happy for it to start at nine and end at five. To loosely tie this in with Matt’s point, it’s about where the line is between profession and passion and the effect of having people around that don’t necessarily care much or are indifferent to what they do.  Personally I find it frustrating and draining. Vigorous debate over things that I truly believe in (software or not) are moments that I live for, so being in situations where that can’t happen is just a little bit soul destroying. The thing with this though is that there are so many levels that you can deliver software on that are all based on the context of the business / project, cost, quality, target audience etc. There is far more work for developers than there are ‘good developers’ to do that work and the fact is that for many situations, near enough just has to be good enough. Personally, that’s just not for me though. Not that I necesarily fall ino this category (yet), but any serious product or company that excels at what they do have no time for that mentality. That’s what sets them apart.

A new release of the week calendar plugin has been published with a number of enhancements and bug fixes. Part of this release included an internal refactoring to base the calendar off the jQuery-UI widget framework. I think this has helped to clean the code up and certainly simplified things. I think there’s still more that can be done in this area and  I expect to refine the internals over the next couple of releases. If anyone’s interested in a run down on the jQuery-UI widget structure, I found this to be a very helpful article.

This release contains the following improvements:

• Added better layout support for overlapping of events (there are still some improvements to be made in this area but it’s a lot better than it was)
• Added ability to supply formatters for dates and times. You can also define your own day, month name arrays to be used by the date and time formatters.
• Added improved demo with creation / editing of events using jquery ui.
• All demos are now served directly out of google code svn
• Added option to only display hours defined in the ‘businessHours’ config option.
• Migrated code-base to extend jquery-ui widget.
• Improved inline method documentation.
• Added ‘readonly’ config option to flag the entire calendar as readonly, preventing creation, dragging, dropping and resizing of events.
• Added ability to configure day, month names for better i18n support
• Fixed bug with IE7 resizing once an event gets to 2 timeslots or smaller
• Added public method for returning an array of valid timeslots  for a given date based on the calendar options. Useful for populating select fields with start and end times.

Thanks to everyone who’s helped submitting bug reports, feature requests. I attempted to get the most pressing ones out in this release and hope to follow it up with another release in the next week or so.