Feb 8 2010

Memcache lockless queue implementation (v2)

Jonathan

The new Google App Engine 1.3.1 SDK has an added method on memcache called “grab_tail”. This is a great little method that removes the need for my lockless queue to manage the read counter. We still need to manage the write counter though, because there doesn’t seem to be a way to add items to a memcache namespace with an arbitrary name.

I have also added a way to rollback in the case that your processing of queue items encounters an error. I was doing the reads non-destructively earlier, so I just needed to reset the read-counter to an appropriate point, but now (because grab_tail is destructive), we need to re-queue the messages.

<rant>

In this implementation, the method that gets the next write counter “__nextCounter”, is far more complicated than it should be, because of an outstanding defect on memcache. This defect is probably quite simple to fix, but since I reported it almost 6 months ago, it hasn’t even been acknowledged. I do really wonder about the value of an issue tracker to a community that doesn’t use it. If the google team are not going to use the issue tracker, it would be far better to acknowledge that and get rid of it completely.

</rant>

Here it is

from google.appengine.api import memcache
import logging
 
class Queue(object):
    itemPrefix="queueItem"
    writeCounter="writeCounter"
    def __init__(self, queueName):
        self.name = queueName
 
    def write(self, msg):
        counter = self.__nextWriteCounter()[0]
        msgKey = self.itemPrefix + str(counter)
        if not memcache.add(msgKey, msg, namespace=self.name):
            raise QueueException("msg key already existed: %s" % msgKey)
        logging.debug("wrote to %s:%s" % (self.name, msgKey))
 
    def writeMulti(self, messages):
        if len(messages) == 0:
            return
        mapping={}
        counters = self.__nextWriteCounter(len(messages))
        for msg,counter in zip(messages,counters):
            mapping[self.itemPrefix + str(counter)]=msg
 
        self.writeMulti(memcache.add_multi(mapping, namespace=self.name))
 
    def read(self):
        result = memcache.grab_tail(item_count=1, namespace=self.name)
        if len(result) > 0:
            return result[0]
        else:
            return None
 
    def readMulti(self, maxItems=100):
        return memcache.grab_tail(item_count=maxItems, namespace=self.name)
 
    def requeueMessages(self, messages):
        self.writeMulti(messages)
 
    def __currentWriteCounter(self):
        return self.__currentCounter(self.writeCounter, 0)
    def __nextWriteCounter(self,howmany=1):
        return self.__nextCounter(self.writeCounter, howmany=howmany)
 
    def __currentCounter(self, key, default):
        counter = memcache.get(key, namespace=self.name)
        if not counter:
            memcache.set(key, default, namespace=self.name)
            counter = default
        return counter
 
    def __nextCounter(self, key, howmany):
        counter = memcache.incr(key, namespace=self.name+"c", delta=howmany)
        if counter is None:
            if not memcache.add(key, howmany, namespace=self.name+"c"):
                # handles the case where another thread got in first with the add
                counter = memcache.incr(key, namespace=self.name+"c",delta=howmany)
                if counter==None:
                    raise QueueException("could not increment counter: %s" % key)
            else:
                counter = howmany
        return range(counter-(howmany-1),counter+1)
 
class QueueException(Exception):
    """Base APIProxy error type."""

and to use it:

queue=queue.Queue(clientViews.QUEUE_NAME)
def processMessages(request):
    msgs=[]
    try:
        msgs = clicksAndViewsQueue.readMulti(maxItems=100)
        if len(msgs) > 0:
            for msg in msgs:
                processMessage(msg)
 
            taskqueue.addTask(url=viewUrl, queueName=taskqueue.BACKGROUND_QUEUE)
    except Exception, e:
        clicksAndViewsQueue.requeueMessages(msgs)
        raise

Oct 14 2009

Money database property for Google App Engine

Jonathan

Calculating money is a tricky thing. Your calculations have to be ultra-precise, so no storing things as floats where $1.33 might actually be stored as 1.3299999999. That is no good for calculations… But equally it is no good storing in cents either: what is 133c /2?

Python has a Decimal datatype, but this requires serialisation to String. Nick Johnson from the Google App Engine team wrote a post about how to write a Decimal property for Google App Engine. Unfortunately this because this serialises to String, then when you do any sorting, then you get String sorting: 100 comes before 11, which comes before 20. Bummer. I played around with storing numbers with a bunch of leading zeroes ie: 0000000010. But that starts to feel a bit hacky.

So I wrote a Money database property that has 6 places of precision, and works as a normal Python numeric type. I haven’t implemented all of those methods, just the ones that I needed. I would be happy to take feedback though.

To use:

class Transaction(db.model):
    dateOccurred = db.DateProperty(auto_now_add=True)
    description = db.StringProperty(required=True)
    amount = MoneyProperty(required=True)
 
t = Transaction(description="I got some money", amount=Money(10.34))
from google.appengine.ext import db
 
class Money(object):
	multiple = 1000000.0
	def __init__(self, val, multiply=True):
		if multiply:
			self._intVal = int(float(val) * self.multiple)
		else:
			self._intVal = int(val)
 
	def format(self, places=2):
		return "%.*f" % (places, float(self))
 
	def __float__(self):
		return self._intVal / self.multiple
 
	def __repr__(self):
		return "%.06f" % (self._intVal / self.multiple)
 
	def __mul__(self, other):
		if type(other) == Money:
 			return Money((self._intVal * other._intVal) / self.multiple, False)
		else:
			return Money(self._intVal * other, False)
 
	__rmul__ = __mul__
	def __add__(self, other):
		if type(other) == Money:
 			return Money(self._intVal + other._intVal, False)
		else:
			return Money(self._intVal + (other * self.multiple), False)
 
	__radd__ = __add__
 
	def __cmp__(self, other):
		if other == None:
			return 1
		elif other == "":
			return 1
		elif type(other) != Money:
			return self._intVal - other*self.multiple
		return self._intVal - other._intVal
 
	def __sub__(self, other): 
		return Money(self._intVal - other._intVal, False)
 
	def __rsub__(self, other): 
		return Money(other._intVal - self._intVal, False)
 
	def __div__(self, other): 
		if type(other) == Money:
			return Money((self._intVal * self.multiple) / other._intVal, False)
		else:
			return Money(self._intVal / other, False)
 
	def __rdiv__(self, other): 
		if type(other) == Money:
			return Money((other._intVal * self.multiple) / self._intVal, False)
		else:
			return Money((other * self.multiple * self.multiple) / self._intVal, False)
 
	def __neg__(self): 
		return Money(self._intVal * -1, False)
 
class MoneyProperty(db.Property):
    data_type = Money
 
    def get_value_for_datastore(self, model_instance):
    	value = super(MoneyProperty, self).get_value_for_datastore(model_instance)
    	if value==None:
    		return None
    	elif isinstance(value, Money):
    		return value._intVal
    	else:
    		return Money(value)._intVal
    def make_value_from_datastore(self, value):
    	if value==None:
    		return None
    	else:
    		return Money(value, False)
 
    def empty(self, value):
    	return value == None
 
	def get_value_for_form(self, instance):
		value = super(MoneyProperty, self).get_value_for_form(instance)
		if not value:
			return None
		if isinstance(value, Money):
			return float(value)
		return value
 
	def make_value_from_form(self, value):
		if not value:
			return []
		if isinstance(value, Money):
			return Money(value)
		return value

Jul 22 2009

The people around you make the difference

rob

Over at 37signals, Matt blogged on a topic recently that really resonated with me… the gist of what he said being that if a project or company is made up of a whole lot of people who don’t really know each other, individuals are generally going to play it safer than a group of people who are comfortable with one another who might fight harder to get their point heard. It doesn’t need to be a shit-fight, just an environment where people can be freely passionate and walk away as friends.  Obviously this is a generalisation and there are always people who will say what they feel – I tend to be one of them although that’s somewhat mood dependent. Anyway, the net effect of this can often be mediocrity which can be damaging or at least limiting to a project or company.

Something else that I find i’m often up against is trying to consider how important someone’s job is to them when I set expectations around quality or general awareness of what’s going on in their professional world. As someone who spends silly amounts of time working on code, reading about development / design etc at all hours of the day, I need to keep reminding myself that for many others it’s just a job and they’re happy for it to start at nine and end at five. To loosely tie this in with Matt’s point, it’s about where the line is between profession and passion and the effect of having people around that don’t necessarily care much or are indifferent to what they do.  Personally I find it frustrating and draining. Vigorous debate over things that I truly believe in (software or not) are moments that I live for, so being in situations where that can’t happen is just a little bit soul destroying. The thing with this though is that there are so many levels that you can deliver software on that are all based on the context of the business / project, cost, quality, target audience etc. There is far more work for developers than there are ‘good developers’ to do that work and the fact is that for many situations, near enough just has to be good enough. Personally, that’s just not for me though. Not that I necesarily fall ino this category (yet), but any serious product or company that excels at what they do have no time for that mentality. That’s what sets them apart.


May 22 2009

Flex Authorize Tag

adam

It is a very rare event when I am asked to code an application which doesn’t have some security constraints. Most Java MVC frameworks provide some type of role based security for the UI out of the box. No such thing exists in Flex. I decided, after casting about for an existing solution, to write an authorize tag for myself. This component has the following properties which need to be set:

  1. roles – one or more roles (strings) which are to be evaluated against.
  2. userRoles – one or more roles(strings) which the user has
  3. components – one or more components which will be effected if evaluation of roles against userRoles fails.
  4. type – type of evaluation to be performed one of:
    • ‘hasAny’ – evaluates true if userRole exists in any of the roles
    • ‘hasAll,’ – evaluates to true only if userRoles contains all roles
    • ‘hasNone’ – evaluates to true if userRoles does not contain any of the roles.
  5. behaviour – the behviour to be applied to the specified components:
    • Disable – disables the components if evaluation fails
    • Vanish – causes the components visible property to be set to false if evaluation fails
    • Collapse – causes the components visible and includeInLayout properties to be set to false if evaluation fails

Here is the code, please feel free to copy it and use it. Any feedback would be appreciated:

package com.express.security {
import flash.events.Event;
import flash.events.EventDispatcher;
 
import mx.collections.ArrayCollection;
import mx.collections.ICollectionView;
import mx.collections.IList;
import mx.collections.ListCollectionView;
import mx.collections.XMLListCollection;
import mx.core.UIComponent;
import mx.events.CollectionEvent;
import mx.events.FlexEvent;
 
public class Authorize extends EventDispatcher{
 
   public static const HAS_ANY : String = "hasAny";
   public static const HAS_ALL : String = "hasAll";
   public static const HAS_NONE : String = "hasNone";
 
   public static const DISABLE : String = "disable";
   public static const VANISH : String = "vanish";
   public static const COLLAPSE : String = "collapse";
   private static const BEHAVIOUR_ENUM : String = DISABLE + "," + VANISH + "," + COLLAPSE;
 
   /**
    * Roles which will be evaluated against the type rules and user's roles.
    */
   private var _roles : ICollectionView;
 
   /**
    * Roles which the current user has. These will be evauated aginst the type rules and roles.
    */
   private var _userRoles : ICollectionView;
 
   /**
    * Components which will have behaviour applied to them based on the evaluation outcome.
    */
   private var _components : ICollectionView;
 
   /**
    * Specifies the type of evaluation which will be applied to the roles
    */
   [Inspectable(enumeration="hasAny,hasAll,hasNone")]
   public var type : String;
 
   /**
    * Specifies the type of evaluation which will be applied to the roles
    */
   [Inspectable(enumeration="disable,vanish,collapse")]
   public var behaviour : String;
 
   public function Authorize() {
      super();
      addEventListener(FlexEvent.CREATION_COMPLETE, handleCreationComplete);
   }
 
   private function handleCreationComplete(event : FlexEvent) : void {
      evaluate();
   }
 
   public function evaluate() : void {
      var result : Boolean = false;
      if (_roles != null && _userRoles != null && type != null) {
         if (type == HAS_ANY) {
            result = evaluateAny();
         }
         else if (type == HAS_ALL) {
            result = evaluateAll();
         }
         else if (type == HAS_NONE) {
            result = evaluateNone();
         }
      }
      applyResult(result);
   }
 
   protected function evaluateAny() : Boolean {
      for each(var userRole : String in _userRoles) {
         if (containsRole(_roles, userRole)) {
            return true;
         }
      }
      return false;
   }
 
   protected function evaluateAll() : Boolean {
      for each(var userRole : String in _userRoles) {
         if (!containsRole(_roles, userRole)) {
            return false;
         }
      }
      return true;
   }
 
   protected function evaluateNone() : Boolean {
      for each(var userRole : String in _userRoles) {
         if (containsRole(_roles, userRole)) {
            return false;
         }
      }
      return true;
   }
 
   protected function containsRole(roles : ICollectionView, role : String) : Boolean {
      for each(var userRole : String in roles) {
         if (role == userRole) {
            return true;
         }
      }
      return false;
   }
 
   protected function applyResult(result : Boolean) : void {
      for each(var comp : UIComponent in _components) {
         switch(behaviour) {
            case DISABLE :
               comp.enabled = result;
               break;
            case VANISH :
               comp.visible = result;
               break;
            case COLLAPSE :
               comp.visible = result;
               comp.includeInLayout = result;
         }
      }
   }
 
   public function get roles():Object {
      return _roles;
   }
 
   public function set roles(val:Object):void {
      _roles = convertToCollection(val);
      evaluate();
   }
 
   public function get userRoles():Object {
      return _userRoles;
   }
 
   public function set userRoles(val:Object):void {
      _userRoles = convertToCollection(val);
      _userRoles.addEventListener(CollectionEvent.COLLECTION_CHANGE, handleCollectionChange, false, 0, true);
      evaluate();
   }
 
   public function get components():Object {
      return _components;
   }
 
   public function set components(val:Object):void {
      _components = convertToCollection(val);
      _components.addEventListener(CollectionEvent.COLLECTION_CHANGE, handleCollectionChange, false, 0, true);
      evaluate();
   }
 
   private function handleCollectionChange(event : Event) : void {
      evaluate();
   }
 
   public function convertToCollection(value : Object) : ICollectionView {
      if (value is Array) {
         return new ArrayCollection(value as Array);
      }
      else if (value is ICollectionView) {
         return ICollectionView(value);
      }
      else if (value is IList) {
         return new ListCollectionView(IList(value));
      }
      else if (value is XMLList) {
         return new XMLListCollection(value as XMLList);
      }
      else {
         // convert it to an array containing this one item
         var tmp:Array = [value];
         return new ArrayCollection(tmp);
      }
   }
}
}

May 14 2009

Simple Object Assembler release update

rob

There have been a couple of releases of the Simple Object Assembler in quick succession over the last week. These changes are all in aid of simplifying it’s use and can dramatically reduce the number of converters required in some projects due to enhancements to the auto-mapping capabilities.  One project i’m using it on saw the converters drop from 34 to 14!

A summary of the release notes from the last week are below:

0.4.1

This is a small api change release that simplifies a couple of commonly used methods. It will require changes to any existing converters. Please see upgrade notes below.

Upgrade notes:

  1. Change all converters that implement the convert(...) method to return void instead of the destination object.
  2. Change all converters that implement the alwaysIgnoreProperties() method to return an IgnoreSet.

0.4.0


May 3 2009

Evolve The Future – Evolution in Javascript

Jonathan

Evolve The Future is a “proof of concept” evolution simulator inspired by Thomas Ray’s Tierra application, which he always promised would be a networked environment for digital wildlife. I haven’t created that, but hopefully this is a step in that direction. Little animals run around, reproduce and eat each other. Each animal is written in a custom assembly-code language running on a number of virtual machines. and all of this is written in Javascript.

I wrote this application a couple of years ago, and kind of forgot about it. But now I have dusted it off, ported it to the Google App Engine and updated it a little bit here and there. I am continuing to work on it, new features (like the ability to look around) will be added soon.

Because javascript isn’t really fast enough (yet) to get enough generations through for evolution to be a useful force before my attention span wanes, this is possibly more of an environment to test your skills in writing in assembly code, and competing against the rest of the assembly-code writing world for the title of … (I’ll think of something :-)   ).

The instruction set is not documented yet, and the animals structure is not documented yet (but both of these soon will be.

Each animal (process) has the following properties

  • Memory: This is both readable and writable memory addresses starting from address 0 for each animal.
  • Threads: Each animal can have multiple threads that communicate via it’s memory.
  • CpuTime: Each animal is granted cputime by it’s parent when it is born, and after that is given cputime by the environment (like sunshine) each cycle.
  • Position: Each animal has a current (xy) position in the grid
  • Direction: Each animal is pointing in a certain direction (this is not currently shown in the display.

Each thread in each animal has the following properties:

  • Stack: this is essentially current memory. Operations can put information on the stack, manipulate the top item on the stack, or pop information off the stack.
  • Execution Pointer: Each thread is executing the same program, but potentially at a different address (different function) in memory.
  • Read Pointer and Write Pointer: These are used for copying operations.
  • Memory Pointer: This is used for writing operands only.
  • Speed: An animal can choose to operate faster, multiple operations per second. This is expensive for the animal.
  • SleepCycles: The thread can sleep for a set number of cycles. It will be woken when this counter reaches 0.

There have so far been two Ancestors created:

  1. A simple non-mobile vegetable. This has a single thread that is devoted to reproduction.
    Once all 8 slots around the animal are full, it sleeps for a period and then checks again.
  2. A simple mobile vegetable (is this a vegetable??). This has two threads: one is devoted to reproduction, the other moves in a straight line forever.
    When it reproduces, it turns right.

Animals reproduce by allocating memory at the end of their memory, then copying code into the new memory, and then dividing the new memory. This creates a new process.
The copying process has a 1 in 1000 random chance of failing to accurately copy the data requested. This results in the child not always being the same as the parent, and the chance for evolution is created.
Successful species (when a species has had enough animals with the same code, occurs at powers of 10) are sent to the server.
When the application starts it requests the 10 best animals from the server.
Things you can do:

  • Click on an individual to see information about it on the “Process Details” tab
  • View code of species on server to see what has evolved from the “Progenitors”
  • Insert new species into the environment
  • Examine the code of any running animal
  • Send me patches!!

Apr 12 2009

Unwanted IE7 Horizontal Scrollbar

rob

I came across an interesting bug today with IE7. Actually, no not interesting, frustrating is a better word to describe it. I had a random appearance of a horizontal scrollbar within a div that has a fixed height set in javascript and overflow auto to achieve a viewport style interface.  The correct  behaviour, the one exhibited in all other browsers, was to add a vertical scrollbar to the fixed height div and not to add a horizontal scrollbar. What IE7 appeared to be doing was not compensating for the vertical scrollbar and adding a horizontal scrollbar with a scroll amount of the vertical scrollbar width.

IE7 horizontal scrollbar being forced by width of vertical scrollbar

IE7 horizontal scrollbar being forced by width of vertical scrollbar

Following ?

Ok, now some searching on the web came up with a lot of mentions of this kind of behaviour in different forms and the common fix seems to be using the following css attributes:

overflow-x: hidden;
overflow-y: auto;

Unfortunately for me this didn’t work. I tried all sorts of variations like forcing widths and a whole bunch of irrational, voodoo style css attributes in the hope that i’d happen across the right combination that would slap IE into submission. Nothing!

Some other behaviour that i’d been thinking was odd but hadn’t given lot of thought to yet was:

  1. Scrolling the scrollbar to the right would cause it to disappear.
  2. Elements such as text, fields etc that appeared underneath the scrollbar showed through
underying elements showing through scrollbar

underlying elements showing through scrollbar

I tried using javascript to simulate a scroll to the right on load – something that made me cringe but it was worth a try. No luck still.

Finally, it occurred to me that the answer could be in the second of the observations described above. Ahaaaa! What if I give the element with the desired vertical scrolling a background color. Given that the elements within the scrollable dive are showing through, maybe giving the element a background color would show through as well and effectively hide the scrollbar. At that stage the element background was transparent so if I give it a white background which is was gaining from an underlying parent element that should cover this rogue scrollbar.  And that was that, the scrollbar is gone.

I immediately tried to describe my frustrations to my girlfriend of one more day of my life being wasted due to Internet Explorer and was somewhat unsatisfied with her lack of understanding.  I’m not sure what i’d expected from her but she definitely appeared to be more interested in the interview with Viggo Mortensen on the television – who’d have guessed!!


Mar 28 2009

Simple Object Assembler

rob

Going back a few years, it took a little while for me to realise the benefits of providing a clean separation of the core business domain from any consuming client, be it a web ui or remote service. I guess I always saw it as a lot of work (which it can be) and the simpler apps I was writing didn’t seem to need it – or at least I couldn’t really see the benefit at the time. While exposing dtos to the outside world is not necessarily a better option in all cases (many modern app frameworks offer rapid development because they don’t do this), i’ve found that erring on the side of caution in most of my development has set me up for a better experience as an application grows, allowing me to develop my domain without worry about how it might be used externally. This approach also has a nice side effect of allowing me to develop from the outside in, giving me exactly what I want in a web client and exactly what I want in my domain… often these are a little different.

I’ve used a few different techniques for converting domain objects to dtos and back again and it’s always turned out to be an ugly part of the application. I gave Dozer a go for a while and while it’s very capable, I found that once you have nested object graphs that need selective conversion for different scenarios, it became more and more difficult to follow, with more chance of things happening that you don’t want. Also having the mapping rules defined in xml, there were too many places to check for any different scenario. It’s been a while since I used Dozer and it’s quite possible that any statements I make about the way it works may be wrong so i’ll just say that it had some things a while back that lead me away from it and I haven’t gone back for another look since. So after trying Dozer out and not being entirely happy, I went back to having a pair of domain / dto assemblers that had dedicated methods for converting one object to another.

For example:

public class DomainAssembler {

public User assembleNewUser(NewUserRequest newUserRequest) {
//… manually convert objects here.
}

}

As you can imagine, this class will get large very quickly so you’ll want to break it down. Depending on the application, it might not be clear where or how much and what the deciding factor is. If you’re using course grained services, maybe separating assemblers on a per-service basis is the way to go. The thing with that is that you end up with lots of inter-dependencies between service based assemblers that is also hard to manage. The worst thing is that you also end up with lots of special methods for converting different parts of a graph. Clear naming is something I always find difficult, probably because I place a lot of importance on it.

So for example, this assembler has a three different methods for returning a different view of a user:

public class DtoAssembler {

public UserDto assemblerBasicUser(User user) {
//… manually convert objects here.
}

public UserDto assemblerUserWithProjects(User user) {
//… manually convert objects here.
}

public UserDto assemblerUserWithProjectsAndTasks(User user) {
//… manually convert objects here.
}

}

Depending on your application, this may or may not be a problem but you can see where it’s going. Anyway, this all bothered me a bit so I started writing something that has helped to simplify things for me a little on a couple of apps. There were a few basic requirements that I wanted to be able to satisfy and some assumptions about the environment that it would exist within. These included:

  • Runs in a Spring 2.5.x based application and it’s ok to have a dependency on this but try to avoid if possible.
  • Likely to be a Hibernate / JPA based application but again don’t tie to the Hibernate API if at all possible.
  • Utilise Convention Over Configuration where possible but don’t impose it.
    • Fields of same names are mapped automatically
    • Fields of same name but different types can be mapped by registering converters for the combination. It’s recursive.
    • Fields of different names can be mapped by explicit configuration
    • Fields to be ignored can be configured
    • Any details about how much of an object graph should be populated must be configurable at runtime
  • Can be completely overridden if necessary or at different stages in the conversion cycle.
  • Configuration information is not defined in xml and is as close to the code that uses it as possible. This being a differentiator from Dozer.
  • Should be able to detect problems in configuration such as mistyped property names.
  • Destination objects should be able to be constructed in any way required but if not defined should be done reflectively.
  • Should be capable of handling conversion of objects proxied by hibernate or JPA.
  • Can handle mapping of collections, using existing objects within the destination collection when it makes sense for integrity or performance reasons.
  • Can handle circular relationships
  • Should use direct field access to negate the need for getters and setters simply for object conversion

Object Assembly / Conversion

The object assembler is the entry point to object conversion. It maintains a registry of ObjectConverters that are written by the application developer to convert one object to another. The registry enables a single converter to exist for one type to another.

You generally call the object assembler like this:

DestinationObject destinationObject objectAssembler.assemble(sourceObject, DestinationClass.class);

This will look up it’s registry for a converter with the same source / destination combination. A converter in it’s simplest form might look something like this:

public class SourceToDestinationTestObjectConverter extends    AbstractObjectConverter<SourceObject, DestinationObject> {

public Class<DestinationObject> getDestinationObjectClass() {
return DestinationObject.class;
}

public Class<SourceObject> getSourceObjectClass() {
return SourceObject.class;
}

}

You’ll notice that this converter extends the AbstractObjectConverter which has most of the conversion logic. The end result of calling this converter via the assembler as shown above is that all properties of the same name will be converted from the source to destination object. If two properties with the same name but different types are encountered, the registry will be consulted and the assembler will recursively call itself until all properties are converted.


Selective Conversion

If you want to exclude any part of the object graph from automatically being converted, you can do so in three ways.

  1. By overriding the ‘disableAutoMapping()’ method on your converter implementation and returning ‘true’
  2. By overriding the ‘alwaysIgnoreProperties()’ method on your converter and returning a Set of field names to never convert
  3. By passing an array or var-args of property names to ignore into the assembler

The first two approaches are useful when you want this behaviour for every invocation of the converter but if you want to be able to specify what to convert upon invocation, the third approach is much more flexible.

Example:

DestinationObject destinationObject objectAssembler.assemble(sourceObject, DestinationClass.class, “ignoreFieldA”, “ignoreFieldB”);

The ignore field paths are aware of nested object graphs so it’s possible to ignore any part of the graph, no matter how deep.

DestinationObject destinationObject objectAssembler.assemble(sourceObject, DestinationClass.class, “ignoreCollection.fieldA”, “ignoreFieldB.fieldValue”);

This approach is very similar to web field binding except that collections are not indexed. This means that while you can refer to a field of an object within a collection, it will always apply to all objects in the collection, not a single one. For example you cannot specify ‘ignoreCollection[0].fieldName’ to ignore only the ‘fieldName’ property of the first element in the ignoreCollection. I can’t think of a situation where this would be that useful. Having said that, adding this capability would be reasonably straight forward.

Wildcard Exclusions

There are times when you want to convert an object that is a property of another but not actually convert any of it’s properties. Say for example, you want to look up a staff member’s manager from the database and populate the manager field in the staff member but you don’t want to actually map any of the ManagerDto fields onto the Manager object from the database. In this case you can use a wildcard in your ignorePath.

StaffMember staffMember objectAssembler.assemble(staffMemberDto, StaffMember.class, “manager.*”);

Mapping fields with different names

Sometimes your source and destination objects don’t have the same names. In this case, it’s possible to define pairs of ’source > destination’ names for mapping. This is done at the converter level by overriding the ‘customConverterFieldMappings()’ method. This method should return a Set of ConverterFieldMappings.

Configuration Validation

It’s easy to mistype a property name so you’ll want some checking to make sure your property names actually exist. Due to the way converters are registered, this can’t be done at startup time yet (hopefully it will soon). It can be done on first execution though so it’s easy to pick this up as part of your test suite. Any fields marked to ignore or fields mapped that don’t exist will result in an exception with a clear indication as to the source of the problem.

Constructing your own Destination Object

The default behaviour is for the converter to attempt to construct a destination object based on it’s destination type. This requires your destination object to have a default no-arg constructor. If you want to be able to construct your own object you can do this by overriding the createDestinationObject(SourceObject sourceObject) method. Because the sourceObject is passed into this method it’s entirely possible to manually do some conversion in here but it’s not the intended place and it’s possible that your values will be overridden during the actual conversion which happens next so it’s not advised. There is a method for this which i’ll come to. This is however a good place to do things like retrieve an object from the database based on the id of the source object. This is certainly the most common case.

Example of a Converter with the createDestinationObject method being used.

@Component
public class UserDtoToUserConverter extends AbstractObjectConverter<UserDto, User> {

private UserDao userDao;

@Autowired
public UserDtoToUserConverter(UserDao userDao) {
super();
this.userDao = userDao;
}

@Override
public User createDestinationObject(UserDto userDto) {
if (userDto.getId() == null) {
return new User();
} else {
return userDao.findById(userDto.getId());
}
}

public Class<UserDto> getSourceObjectClass() {
return UserDto.class;
}

public Class<User> getDestinationObjectClass() {
return User.class;
}

}


Custom Conversion Logic

There are cases where you need to perform some custom conversion logic because it’s not simple enough to just map one field to another. In this case you can override the convert(SourceObject sourceObject, DestinationObject destinationObject) method. This is called after all auto conversion has been performed and enables any custom conversion to be done. If you wanted to perform a full custom conversion for a converter you simply override disableAutoMapping() to return true and write your custom logic here.

Conclusion

This is not a perfect solution and it certainly won’t make sense in all situations. Some people, justifiably will take issue with the idea of passing strings of property fields to ignore / exclude into the assembler. I find this a bit awkward myself however i’m not sure that it’s really all that different to putting it outside in an xml file. I feel this line is starting to be crossed with some implementations of annotations vs their xml configuration approaches. Sure, it’s compiled code, but it’s also a lot more visible at the point at which it matters so there are some clarity benefits there. It would also be possible to externalise this configuration as an option but the more that’s done, the closer we get to Dozer which undeniably has a much deeper feature set and is almost certainly much ’smarter’ under the hood with the benefit of years in the wild. The intention for this was a simple solution to a simple mapping problem, sitting halfway between dozer and a fully handwritten approach. I came across an interesting post the other day by Gavin King, talking about the new typesafe Criteria API for JPA 2.0. I’ve been pondering whether one of the features they are using from the Java 6 compiler plugin architecture might be useful here although introducing yet another annotation could become a bit of a bore!

Another potential problem comes about when new properties are introduced to objects that are already mapped. It’s possible that an introduced relationship with another large part of the object graph could go unnoticed and result in conversion of much larger sets of objects than originally intended. I guess this is a trade off of taking blacklist vs whitelist approach for property mapping.

Get the Code

I’ve published the code on google code. At this point it’s not packaged up so you’ll have to check it out and build it from the source.  It’s a maven project though so this will be trivial as long as you’re familiar with maven.


Feb 25 2009

Integrating Spring Security with BlazeDS and Flex RIAs

adam

I have found a couple of tutorials on this lying around the web, however they all take slightly different approaches, and although the information was useful, I think there is a cleaner way to do the integration. Keeping in mind of course that now that the spring guys have released the spring-flex module there will hopefully be complete integration with Spring Security soon without the need to even wire it up the way I describe here.

What I am going to describe here uses:

  • Spring 2.5.5
  • Spring-Security 2.0.4 (core & tiger)
  • BlazeDS 3.2
  • Flex3

My approach is not to protect end-points using BlazeDs’s security. The reason is that I often expose my services as web services as well as AMF endpoints and I don’t want to create my security mappings twice. Instead I use method invocation based security and protect the services directly. All that BlazeDS needs to do is get the users credentials out of the request and delegate to SpingSecurity to create a SecureContext, authenticate and put the authentication object in the SecureContext (effectively thread local). After that normal Spring Security applies and secure methods will be protected in the normal way.

A typical security scenario in one of my applications goes something like this: Either none of the application or a small part of the application is available anonymously as is the login service. Users access them without credentials, usually the User doesn’t even have menu options to navigate to UI areas with secure functions until they have logged in. After a successful login I will cache the user’s credentials on the client and re-log them in on each service request. This keeps the server completely stateless, all state is stored on the client. I configure a lightweight cache on the server since if someone logs in once they will likely make many more login requests (one for every service they are going to use). This reduces database hits. I implement a service registry on the client which stores the credentials and automatically adds them to all RemoteObjects and HttpServices.

I’m going to work this assuming that you already have a Flex/BlazeDS/Spring application going. It can be as simple as hello world, there is an excellent article on how to do this using the new spring-flex solution here . The way that i am applying security means that it is largely irrelevent whether you are using the Adobe SpringFactory or the spring-flex MessageBrokerHandlerAdapter.

Step1. Add spring-security to your web application’s lib my pom entry looks like this:

<dependency>

   <groupId>org.springframework.security</groupId>

   <artifactId>spring-security-core</artifactId>

   <version>2.0.4</version>

   </dependency>

<dependency>

<groupId>org.springframework.security</groupId>

<artifactId>spring-security-core-tiger</artifactId>

<version>2.0.4</version>

</dependency>

   </dependencies>

Step2. Add the spring security filter to your web.xml

<filter>

<filter-name>springSecurityFilterChain</filter-name>

<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>

</filter>

<filter-mapping>

<filter-name>springSecurityFilterChain</filter-name>

<url-pattern>/data/messagebroker/*</url-pattern>

</filter-mapping>

This will secure everything and is mostly necessary if you are going to use other transports than just AMF via BlazeDS.
Step3. Configure Spring Security. I am just going to put a very basic config here. For details on configuring spring security you can look at the spring security documentation here

<security:http auto-config=“true”>

<security:intercept-url pattern=“/**” filters=“none”/>

</security:http>

<security:global-method-security>

<security:protect-pointcut

expression=“execution(* net.oneadam.app.services.*Manager.*(..))”

access=“ROLE_ADMIN”/>

</security:global-method-security>

<authentication-provider>

   <user-service>

<user name=”adam” password=”adamspassword” authorities=”ROLE_USER, ROLE_ADMIN” />

   <user name=”bob” password=”bobspassword” authorities=”ROLE_USER” />

   </user-service>

  </authentication-provider>

This configuration passes everything through at the URL level and relies on method level security. I have used pointcuts here since it doesn’t require me to cut and paste classes and methods in order to demonstrate how annotations work. The above pointcut requires the user for all methods in any class ending in the word Manager in the services package to have the role ROLE_ADMIN.
Step4. Implement the flex.messaging.security.LoginCommand Interface so that the doAuthentication creates a UsernamePasswordAuthenticationToken and authenticates using Spring Security, if it is successful it should put the authentication object into the SecureContext where it will be accessible to your application and the Spring Security annotations or security pointcuts. here is my implementation:

package net.oneadam.web.security;

import java.security.Principal;

import java.util.List;

import java.util.Map;

import javax.servlet.ServletConfig;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.springframework.context.ApplicationContext;

import org.springframework.security.AbstractAuthenticationManager;

import org.springframework.security.Authentication;

import org.springframework.security.context.SecurityContextHolder;

import org.springframework.security.providers.AbstractAuthenticationToken;

import org.springframework.security.providers.UsernamePasswordAuthenticationToken;

import org.springframework.web.context.support.WebApplicationContextUtils;

import flex.messaging.FlexContext;

import flex.messaging.io.MessageIOConstants;

import flex.messaging.security.LoginCommand;

public class SpringSecurityLoginCommand implements LoginCommand {

   private static final Log LOG = LogFactory.getLog(SpringSecurityLoginCommand.class);

   private AbstractAuthenticationManager authenticationManager;

   public Principal doAuthentication(String username, Object credentials) {

String password = extractPassword(credentials);

if (password == null) {

   return null;

}

  

if(authenticationManager == null) {

   init();

}

Authentication auth = new UsernamePasswordAuthenticationToken(username, password);

try {

   SecurityContextHolder.getContext().setAuthentication(authenticationManager.authenticate(auth));

   LOG.info(“User [" + username + "] logged in.”);

   return (AbstractAuthenticationToken) SecurityContextHolder.getContext().getAuthentication();

}

catch (RuntimeException e) {

   LOG.info(“User [" + username + "] failed authentication.”);

   throw e;

}

   }

   /**

* We are not going to perform Authorization at this level since we would like SpringSecurity

* to manage Authorization at the method level.

*/

   @SuppressWarnings(“unchecked”)

   public boolean doAuthorization(Principal principal, List roles) {

return true;

   }

   public boolean logout(Principal principal) {

HttpServletRequest request = FlexContext.getHttpRequest();

if (request != null && request.getSession(false) != null) {

   try {

request.getSession().invalidate();

   }

   catch (IllegalStateException e) {

// Expected.

   }

}

request.getSession(true); // Session re-created to avoid Flex error when

// it also attempts to invalidate the session.

LOG.info(“User [" + principal.getName() + "] logged out.”);

return true;

   }

   public void start(ServletConfig config) {

//My tests show this never gets called

   }

   public void stop() {

authenticationManager = null;

   }

   @SuppressWarnings(“unchecked”)

   protected String extractPassword(Object credentials)

   {

   String password = null;

   if (credentials instanceof String)

   {

   password = (String)credentials;

   }

   else if (credentials instanceof Map)

   {

   password = (String)((Map)credentials).get(MessageIOConstants.SECURITY_CREDENTIALS);

   }

   return password;

   }

   private void init() {

HttpServletRequest request = FlexContext.getHttpRequest();

ApplicationContext ctx =

   WebApplicationContextUtils.getWebApplicationContext(request.getSession().getServletContext());

authenticationManager = (AbstractAuthenticationManager) ctx.getBean(“_authenticationManager”);

   }

}

You will notice that the doAuthentication method just returns true. That is because I am not interested in securing BlazeDS destinations. I prefer to secure the service facade methods themselves, that way if I expose them using CXF or some other Web Service transport my security does not have to be redone.
Step5. Wire up security in the services-config.xml file to use your SpringSecurityLoginCommand it will look something like this:

<security>

<login-command class=“com.renewtek.cmf.web.security.SpringSecurityLoginCommand” server=“Tomcat”>

   <per-client-authentication>true</per-client-authentication>

</login-command>

<security-constraint id=“roleuser”>

   <auth-method>Custom</auth-method>

   <roles>

<role>ROLE_USER</role>

<role>ROLE_ADMIN</role>

   </roles>

</security-constraint>

   </security>

The roles are actually unnecessary since I return true from all authorization requests at this level. I have just left them in for completeness. If you wanted to reject a request at this level that is where you would put the roles and if you are using the Adobe style of defining destinations in your remoting-config.xml file then you could say what destinations are allowed for what roles. What is important to the strategy I described above is the per-client-authentication entry. it needs to be set to true since we don’t want to use server based sessions. it is false by default.


Jan 29 2009

javascript closures are fun

Jonathan

I really like closures in javascript. I really miss them when I am doing work in Java,  but sometimes when I use them I get a little messed up. Take the following example:

var i;
var objects=[1,2];
for (i=0;i < objects.length;i++) {
    var callback=function() {
        console.log(objects[i]);
    };
    doSomethingAsynchronous(callback);
}

Then you would might expect “1″ and “2″ to be logged when the callbacks come back. This is wrong, instead “2″ and “2″ are logged.

Due to the nature of closures, “i” is one of variables that fall into the scope of the closure by reference, not by value. So, when the value of “i” is updated in the for loop, the current (latest) value is then referenced later on by the closures. So the last item in the list is the item that is accessed.

The way to get around this is to modify the above code slightly, to make the item that you want an argument of a function, rather than accessed through closure scope. Eg:

var i;
var objects=[1,2];
function doSomethingAsynchronousAndLogIt(value) {
    var callback=function() {
        console.log(value);
    };
    doSomethingAsynchronous(callback)
}
for (i=0;i < objects.length;i++) {
    doSomethingAsynchronousAndLogIt(objects[i]);
}