redredred

When writing your application on Google App Engine, you are inevitably going to deploy a version to production that does not have a final set of features. This is (of course) unavoidable. So, then for version 2, when creating new features you will also (probably) have to refactor the data model, at least adding new fields, and potentially renaming fields, or creating composed entities. So, when putting version 2 into production the data that all of your users have created in version 1 will need to be migrated to the new data-model.

Nothing exists as of now to do something similar to Rails Migrations, so everything pretty much needs to be done yourself.

The solution that I outline here is an automated way of processing over all entities in a list of Model classes. This solution loads the entities up into the v2 model classes and then calls your code to modify them (and create any new objects that might be required). This solution will not work for the case where the v2 model is missing fields that need to be read to migrate to the new state. That solution would need to be written using the underlying Entity classes and Query. (I haven’t needed to do that yet). This is really more of a recipe that can be modified to your purposes, than a generic drop-in migration tool.

The idea came from code copied from a google demo.

Here is the file: migrate.py

Directions for use (these directions are for Django, but if you are using the basic handler it would look very similar):

1. Rename migrate.py to migrateXToY.py (for your X and your Y)

2. Modify your urls.py to contain:

patterns("migrateXToY",
                       (r'^migrate/migrateXToY$', 'migrateXToY'),
                       (r'^worker/migrateXToY/(?P[^/]+)$', 'migrateModel'),
)

3. Create a migrateXToY method in the migrate.py like the following (modelsToMigrate is the list of model names that will be processed):

def migrate1To2(request):
    modelsToMigrate = ['User', 'Foo', 'Bar', 'BarDetail', 'Image', 'Report']
    [taskqueue.add(url='/worker/migrateXToY/%s' % i) for i in modelsToMigrate]
    return HttpResponse("created all tasks for processing")

4. Create a FooWorker class like the following (the migrateItem method will receive each item that is being migrated, the method should return the item after processing and should not put it. For efficiency puts are batched):

class PlacementWorker(MigrationWorker):
    kind=Foo
    kindName="Foo"
    def migrateItem(self, item):
        logging.info("processing %s %s" % (self.kindName, item.key()))
        #whatever processing you need to do
        item.cancelled=False
        return item

I had a need for an application that I am writing on Google App Engine for a way to store jobs and then process them all at once. I found the idea for a Memcached lockless queue and created an implementation of it: queue.py
To write to it (I do this from a view where I want to store some stats about the data that was shown in the view):

thisQueue=queue.Queue(QUEUE_NAME)
def method():
    thisQueue.write(data)

and then later on to read from it. I created a cron job that is executed often.

thisQueue=queue.Queue(QUEUE_NAME)
def cronMethod():
    msg = thisQueue.read()
    while msg:
        processMessage(msg)
        msg = thisQueue.read()

If you are using this on Google App Engine, you should also be careful that you don’t run out of time to execute. If you get a DeadlineExceededError after you have read, but before you have finished processing, then the message might get lost.

A new release of the week calendar plugin has been published with a number of enhancements and bug fixes. Part of this release included an internal refactoring to base the calendar off the jQuery-UI widget framework. I think this has helped to clean the code up and certainly simplified things. I think there’s still more that can be done in this area and  I expect to refine the internals over the next couple of releases. If anyone’s interested in a run down on the jQuery-UI widget structure, I found this to be a very helpful article.

This release contains the following improvements:

• Added better layout support for overlapping of events (there are still some improvements to be made in this area but it’s a lot better than it was)
• Added ability to supply formatters for dates and times. You can also define your own day, month name arrays to be used by the date and time formatters.
• Added improved demo with creation / editing of events using jquery ui.
• All demos are now served directly out of google code svn
• Added option to only display hours defined in the ‘businessHours’ config option.
• Migrated code-base to extend jquery-ui widget.
• Improved inline method documentation.
• Added ‘readonly’ config option to flag the entire calendar as readonly, preventing creation, dragging, dropping and resizing of events.
• Added ability to configure day, month names for better i18n support
• Fixed bug with IE7 resizing once an event gets to 2 timeslots or smaller
• Added public method for returning an array of valid timeslots  for a given date based on the calendar options. Useful for populating select fields with start and end times.

Thanks to everyone who’s helped submitting bug reports, feature requests. I attempted to get the most pressing ones out in this release and hope to follow it up with another release in the next week or so.