Rightly or wrongly I get the impression that most RoR engineers ignore database views and would rather they didn’t exist. I’m neutral about db views but there are times when they have their place. We use them in Zico our online survey app for some of the survey stats and gamification we provide.

Last week I was baking some of this is to the app and realised whilst it is simple enough to map a model on to a db view by just treating the view like a table, it was a different matter when it came to testing. Important note: you can only “treat db views like a table” if it is a single-table view or your model is read-only].

The problem with testing views is that when you create your test database, Rails creates the view as a table in the test database. In most circumstances this means your test will fail as your view-now-table won’t have the data in it (the data is selected with a db view as opposed to inserted ). This fails both for test dbs created through migration and structure cloning. Annoying!

I thought of several ways around it (confession: including removing the tests) but looking harder saw that it was possible to copy the view to test db as a view with a bit of monkey-patching. This example only covers MySQL but the adapter classes are there for all the other dbs and once you know how to pull metadata from your db it’s pretty easy.

First … how Rails copies your database structure to test

There are two ways in which Rails copies the db structure over to test. It either runs the migrations on the test database (ignoring the execute sql ones that create db views — #fail) or it copies the structure from SQL (and in this case it turns views in to tables — #fail)

We are going to choose the second way i.e. to clone the database and change the cloning (for MySQL dbs) so that it includes views.

First off we need to tell Rails to clone the database from SQL and we do this by adding this config.active_record.schema_format = :sql to our application.rb file (or similar in Rails2)

You then use the rake db:test:clone_structure command to copy your database structure (i.e. not data these will be via your fixtures) from dev to test.

Next … how Rails maps models

As you might imagine, Rails gets the info need to based models on tables from the database metadata – i.e. info about the tables and columns amongst other things. Every RDBMS has this and you can read about MySQL metadata in the manual.

By default Rails doesn’t pull metadata info about db views (for good reason as unless a single-table view you won’t be able to create or update). However, if your only have the need for read-only models for your views then you can get Rails to do the hard-work for you rather than write the model by hand.

The metadata queries can be found in a set of classes that inherit from ConnectionAdapters::AbstractAdapter. The Rails3 adapter is called Mysql2Adapter and is in the mysql2 gem.

This class defines a method called structure_dump which queries the metadata tables and pulls back all matching tables. We need to re-open that class and override the method to pull back both tables and views.

I added the following to a new file that I put in the config/initializers folder (put in your environment in Rails 2)

support_db_views.rb

module ConnectionAdapters

class Mysql2Adapter < AbstractAdapter
# SCHEMA STATEMENTS ========
def structure_dump
if supports_views?
sql = “SHOW FULL TABLES WHERE Table_type IN (‘BASE TABLE’,’VIEW’)”
else
sql = “SHOW TABLES”
end
select_all(sql).inject("") do |structure, table|
view = table‘Table_type’ == “VIEW”
table.delete(’Table_type’)
if view
structure += select_one(“SHOW CREATE VIEW #{quote_table_name(table.to_a.first.last)}”)“Create View” + “;\n\n”
else
structure += select_one(“SHOW CREATE TABLE #{quote_table_name(table.to_a.first.last)}”)“Create Table” + “;\n\n”
end
end
end
end
end

Basically the method used to read through the metadata and pulled the structure definition (i.e. the CREATE statements) for tables and then collected these for running in to test. We have just changed the metadata query to include views as well as tables — sql = “SHOW FULL TABLES WHERE Table_type IN (‘BASE TABLE’,’VIEW’)” — and then when we loop through the objects, if we find a view we pull its create statement. Simple!

As we’re re-opening our abstract class we need to redefine the connection method that it provides. I won’t put the code in the post but you can download a copy here

Finally … was it worth it?!

As a rule I tend to avoid db views for new Rails apps in that it tends to make life a bit more difficult. There are good reasons to use db views though and if you want (or have) to do use them and you want to write full tests (which of course you do) then this isn’t a bad way to support it.

We’re using paperclip to store files to S3 on a rails 3 project.

Our buckets are in europe and the s3 gem default sets the host to the US.

Here’s a quick initializer to override and set to use a european host (or one of your choosing).

Code from ella with minor adjustments:
https://github.com/marcel/aws-s3/issues/4

aws_s3_europe.rb

 
module AWS
  module S3
    ENDPOINTS = {
      :us     => 's3-us-west-1.amazonaws.com', 
      :asia   => 's3-ap-southeast-1.amazonaws.com', 
      :europe => 's3-eu-west-1.amazonaws.com'
    }
    
    class Base
      class << self
        def region= region
          endpoint = ENDPOINTS[region] or 
          raise S3Exception, "Unknown region: please use one of #{ENDPOINTS.keys.map(&:inspect).join(', ')}."
          DEFAULT_HOST.replace endpoint
        end
      end
    end
  end
end

AWS::S3::Base.region = :europe

Recently we’ve seen a growing number requests from clients and potential clients to help tame their data mountain to get new insights and also report on day-to-day business activity.

We’ve used Infobright and latterly Hadoop to achieve this and have been able to serve up some good results quickly using both tools.

One slight headache has been the learning curve in setting up and configuring Hadoop. We’ve made our own Hadoop AMIs which we can spin up on Amazon Web Services but there’s still a little bit of work to do in manually setting up job configurations, spinning
up servers etc.

Amazon provide their own managed Hadoop service, Elastic MapReduce which takes away a fair bit of the pain for you although at a slightly increased cost. We got out our calculators and figured that the time saved was worth the money as it meant we would get to the business issues earlier.

This post is a brief intro to setting up an example Elastic MapReduce job based on the UFO sightings dataset distributed by Infochimps (we’re using the TSV file if you want to reproduce).

There’s a perfectly good
and well-written intro on the AWS site so if you’re just
getting started then best read that first as this post dives in to an example.

The UFO dataset is our data “Hello World!” and we already have several Hadoop Streaming tasks that manipulate this.

In true example style, the one here is the simplest – it takes the sightings since 1900 and counts them by year.

There are four steps to running the job on AWS:

1. Upload your data to S3: the input file, any mappers or reducers
2. Create a new job flow (using the Elastic MapReduce CLI – see instructions in the AWS guide above)
3. Check the job has run ok and is terminated (otherwise you’ll be charged as it runs)
4. Download your data for further processing

Upload data

This speaks for itself. There are only a couple of things to note really. First, make sure that your output folder doesn’t already exist in S3 otherwise Hadoop will fail. Secondly, you can have different buckets in S3 for any of the items. The job flow script specifies where these all live.

Below are snippets from the map and reduce jobs.

Mapper – map_yyyy.rb

POS_SIGHTING_YEAR=0
CUTOFF_YEAR = 1900


STDIN.each_line do |line|
tokens=line.split("\t")  sighting_year=tokens[POS_SIGHTING_YEAR].slice(0,4)
puts "#{sighting_year}" if sighting_year.to_i > CUTOFF_YEAR
end

Reducer – reduce_yyyy.rb

last_year,count=nil,0
puts "year,count"
STDIN.each_line do |year|
year.chomp!
if last_year && last_year != year
  puts "#{last_year},#count}"
  count = 1			
else				
  count+=1			
end
last_year = year  
end
puts "#{last_year},#{count}"

Create a new job flow

This is done via the AWS Elastic MapReduce Command Line Interface (CLI).

Instructions on how to install and use are in the link above but the syntax is pretty self-explanatory:

ruby elastic-mapreduce --create --name "UFO sightings by year" --stream --mapper s3://<path to your bucket>/map_yyyy.rb --reducer s3://<path to your bucket>/reduce_yyyy.rb --input s3://<path to your bucket>/ufo_awesome.tsv --output s3://<path to your bucket>/output

The key points to note are the fact it’s a streaming Hadoop job, and the fact that the output location can be called whatever you want – just make sure it doesn’t already exist.

After running this, it will create a new Elastic MapReduce job on an EC2 instance (by default a small one), run the job using the mapper and reducer specified and store the output in your S3 bucket.

You can track the progress of the job (and terminate if necessary) in the AWS Management Console on the Elastic MapReduce tab.

Download the data!

All that’s left to do is download the data from the location you specified and do any additional processing on it. You’ll notice there’s also a folder in your S3 bucket with job statistics which can be useful when tuning a job.

Here’s a slightly more complex visualisation of the data where we show sightings by US state, year, and month

Happy mapping (and reducing)!

I had a strange error trying to connect to AWS using Filezilla. We use bitnami images, connecting using PuTTY with Pageant to manage keys.

Each time I tried to connect, with Pageant running, I got the error message “Disconnected: No supported authentication methods available” but colleagues with the same settings didn’t get that. I didn’t have any problem connecting via PuTTY.

I found the solution via this ticket report. I ran the reg edit and deleted my PuTTY Session entries and the connection issues went away.

We’re going to be publishing a series of articles looking at the role of metadata plays in enterprise environments.

Metadata spans all aspects of an enterprise, acting not only as the glue between information but also at the heart of determining what is relevant to the end-user in a given scenario. As such, a holistic approach to metadata is needed to maximise your organisation’s chance of success.

We’re going to be looking in to typical challenges enterprises face:

• Ownership – who is responsible for and who manages the metadata
• Mobility – Making sure that knowledge captured in one area can be found and accessed easily in another
• Lifecycles – Metadata and business lifecycles and how to align them
• Technical – Some of the technical challenges in managing metadata

We’ll also be looking at metadata along some different dimensions. For example:

• Time-based – Enterprises typically work hard to establish a corporate taxonomy but that only caters for what has happened in the past. Consider organisations where new products and services are constantly being created. To maximise value, information must be shared quickly with the rest of the organisation. These “hot topics” won’t appear in any taxonomy. We’ll look at the complete metadata lifecycle from emergence through management to end of life.
• Structured / Unstructured – Metadata is really the glue between your structured and unstructured information. It’s easy to overlook the massive value that can be created by simply linking these two worlds together. For example, imagine you were engaged with a potential client who was a Tier 1 Investment Bank and wanted to find out all of the services and experts that I should be bringing to that conversation.
• Global / Local – Knowledge sharing in international organisations is complex but getting it right means much greater rewards. Whilst there needs to be consistency to allow for sharing across territories it’s important to allow countries to express what is important to them at a national level. For example, consider writing a proposal to support an international firm, how would you find case studies and experts across a multi-national, accessing local information classified in local language.

We’ll be walking through these challenges and considering the business, governance and technical challenges as we go.

After getting a mail from London 2012 telling me that they would tell me what tickets we’d got soon I finally broke and decided that I needed to work out what tickets we’d got now!

I knew how much we’d been charged and the events that we’d applied for. Also, it seems that there is a 6 quid charge to deliver the tickets (I’m expecting my postman dressed as a traffic cop for that) so that means it was just a case of working out which combinations of tickets add up to that.

This is another example of where Ruby makes something that could be tedious or complicated nice and easy by having methods on their collections that make this a breeze – namely Array.combinations(1) where you can do “n Choose p” calculations and also Array.inject which makes summing easy. Data lines aside there’s probably more lines of comments than of code.

The outcome was just a single combination (there are either-ors as some of events are priced the same):

• Athletics
• Volleyball or Beach Volleyball
• Rowing or Basketball
• Football

which is good news as I was really hoping for athletics (and beach volleyball)! I’ve looked through the code a few times and can’t find any issues but any errors gratefully received before I get too excited!

# olympic tickets helper

events={}
# populate a hash of event prices and the event(s)
# 
events[160]="(volley or beach)"
events[500]="athletics"
events[380]="athletics"
events[120]="football"
events[200]="cycling"
events[220]="bball"
events[260]="athletics"
events[134]="(rowing or bball)"

# pull the prices out
prices=events.keys

# the total taken from the credit card
total = 920

# according to the website 6 quid is taken off for delivery (what a rip-off!)
delivery_charge = 6

# the price of the tickets is the remainder
remainder = total - delivery_charge

# 
# this loop controls the number of combinations from the 8 price events
# we know that no single event is over the remainder so start from 2. we also know
# that all 8 are over the remainder (as probably are 7 but too tiresome to work out) so
# only need to work out from 8 choose 2 to 8 choose 7 
#
for i in (2 .. 7)

	# prices choose i ... turn to an array to check each
	possibles = prices.combination(i).to_a

	# for each of the combinations
	possibles.each do |posse|

		# ... find the sum of the prices and check whether they match the remainder
		if (posse.inject(0) {|sum,v| sum+v}) == remainder

			# if they do then they are a combo (possible cos there are multiple events at the same price)
			puts "Possible combo:"
			posse.each do |p|
			 puts "#{events[p]} @ #{p}"
			end

		end #end if

	end # end inner
	
end # end outer

# don't forget the delivery charge
puts "delivery @ 6"

Update 20/7/2011

We ended up with Athletics, Volleyball, Rowing, and Football. So good from a code point of view not so good as I’d convinced myself that we’d be going to the basketball! Still very lucky to get the tickets and really looking forward to it!

We have been migrating a lot of our apps onto JRuby with a goal of running them on Tomcat on AWS.

One of difficulties in doing that is that we can’t run migrations but need to migrate our database in just the same way you would for a standard rails app.

I searched around assuming there must be a solution and found Jay Fields article Rails Generate SQL from Migrations and also a hint at a tool active_record_io_mode which I could no longer access the source code for.

I had problems making Jay’s code run for certain migrations and not being as smart as him couldn’t fix them so I created a more basic version which is a combination of Jay’s and Mark/Zach’s approach.

I have patched Active Record to do a few things:

• Patch ActiveRecord::Base to store an object which any sql that’s run gets written to
• Patch the AbstractAdapter log_info method to check for that object and write any sql to it (with appropriate separators)
• Patch the Migrator migrate method to create an up or down file for a migration and set that in the object (step 1).

This file just needs placing in lib at the start of your project and then require it in your environment.rb. I’m sure there is a cleaner way to do this.

There are a few limitations of the approach:

• you have to run a migration to get the sql (which really means a migrate/rollback/migrate)
• it’s not been tested far and wide, only on 2.3.5 and using jruby

Still, it helps us and I’m sure we can finess it as we move forward.

I’m currently working on a client project where we’re making web service calls via JAX-RPC. Web Services are hugely important and useful but I can’t be the only person to think they’re a royal pain to develop and deploy!

If you’re using Apache Axis then there are some inbuilt tools that can help work out what’s going wrong when things go wrong. The most useful is tcpmon which acts as an “app-in-the-middle” and displays your SOAP request and response.

Here’s a quick how-to in setting up tcpmon in order to intercept your development web services calls and responses. Once you start using it the rest is pretty obvious but I’ve also included some screenshots of the input and output just so you can see.

The instructions assume Windows but replace the command prompts and environment variables with the usual in *nix and it all should work the same.

1. Open a command prompt and set the Java classpath to pick up the Axis classes (tcpmon is bundled with them)
2. set CLASSPATH=CLASSPATH;[my axis classpath]/wsif.jar;[my axis classpath]/axis-1.4.jar;[my axis classpath]/axis-ant-1.4.jar;[my axis classpath]/commons-logging-1.0.4.jar;[my axis classpath]/commons-discovery-0.2.jar;[my axis classpath]/wsdl4j-1.5.1.jar
3. Start tcpmon java org.apache.axis.utils.tcpmon
4. Enter the port where you want to run tcpmon (in the example below I use 1976 but check with netstat to find a free port)
5. Make the webservice call from your test client (the code snippet below is from a JUnit test method)

	// Note that the port is the tcpmon port and it will be forwarded to the web service port
	String endpoint = "http://localhost:1976/axis/services/PortingPort";

	Service service = new Service();
	Call call = (Call) service.createCall();

	// Prepare the call for the Web service
	call.setTargetEndpointAddress(new java.net.URL(endpoint));
	call.setOperationName("myService");
	MyParameterRequestType por = new MyParameterRequestType();
	por.setGreeting("Hello");
	call.addParameter("requestParam",new QName("http://mywsdl.endpoint.for.service/Service"),ParameterMode.IN);
	call.setReturnType(new QName("myParameterResponse"));
	call.invoke(new Object[] {por});

Below is the tcpmon “console” where you set the port that tcpmon listen on.

… and this is the request and response intercepted by tcpmon after making the test call

Last week we ran up a JRuby contacts tracking app which we deployed on AWS. It’s small but beautifully-formed – insert your own joke now – and is a lot better than our having to exchange emails with contacts and leads.

Despite that, yesterday I got fed up of copying contacts from LinkedIn to the app so I thought I’d add a way the user can browse and add their LinkedIn contacts. This was straightforward enough to do in a day, but also had a few gotchas along the way so after a quick bit of background I thought I’d share those.

LinkedIn provide both a JS and a REST API to their
services and split the API in to three primary domains:

• Profile API – to get information about people (e.g. name, role, company, etc.)
• Connections – provides a list of connections for a user
• People Search API – which mimics the search capabilities when you log in to LinkedIn

There are a couple of other APIs around Invitations and Sharing updates etc but those were less of interest to what we were trying to achieve.

Our goal was pretty simple – to save data entry by accessing our connections from LinkedIn and copying their name, role, company, and LinkedIn profile to our contacts application.

This broke down in to 5 technical tasks:

1. Register our contacts application with LinkedIn (needed for the authentication
   step)
2. Authenticate using OAuth and the tokens provided from registration
3. Retrieve connections
4. Pull profile information from each connection
5. Save contact information in to the contacts database

LinkedIn uses OAuth to authenticate and authorize and having had fun in the past with OAuth and Google I was glad this time to see that Wynn’s API not only wrapped OAuth but also had some examples that worked
out of the box.

I’m not going to replicate the code here so I recommend people take a look.

With authentication, the only issue I had was with the OAuth gem under JRuby. It was throwing an exception “Wrong # of arguments (3 for 2)” in the digest.rb class. All that it needed was a different Digest object to be instantiated. I put the following code in to a module called digest.rb and all was well:

require 'oauth/signature/base'
require 'digest/hmac'
require 'openssl'
#
# Hack/fix to allow oauth to be used with JRuby / Tomcat.
# The digest class doesn't work so use the OpenSSL digest class instead
#
#
module OAuth::Signature::HMAC
  class Base < OAuth::Signature::Base
    private
      def digest
        self.class.digest_class Object.module_eval("::Digest::#{self.class.digest_klass}")
        digest  = OpenSSL::Digest::Digest.new('sha1')
        OpenSSL::HMAC.digest(digest, secret, signature_base_string)
      end
   end
end

Once authenticated then it’s just a case of using the methods in the gem’s
Client class to fetch back the objects. Not all of the API is supported but there was enough for what we needed to do. Again, the detail is in the gem source and examples, and the only more “complex” thing that we did was to take the “headline” field (e.g. my headline is Co-Founder at True North) and split that in to a role and position.

# most are --role-- at --company-- but will not fit all
posn_company = lic.headline.split(" at ")
posn=posn_company[0]
company=posn_company[1]||""

Not everybody’s is in this format but enough are to make it not worth going any more complex.

Finally it was a case of deploying on to our running version of contacts on AWS. We used a modified Ubuntu/Tomcat/MySQL bitnami stack again as it’s been good to us before and is pretty straightforward to set up.

This is where the final gotcha
got me. The version of Ubuntu didn’t have some of the XML libraries needed and I ended up with “Could not open library xml2 libxml2.so”. This was just a case of installing those libraries on the machine.

Again, the joy of AWS and the bitnami AMI we could just run
sudo apt-get install libxml2 libxml2-dev libxslt1-dev. I made a us a new AWS AMI with the patch in (so I don’t have to remember again).

For us that was it. The code itself was quick to do – some of the issues took time but by the end of the day we were all happily importing from LinkedIn.

Ping me on http://twitter.com/truenorth_sid if you want to know more or are interested in a cloud-based contacts tracker with LinkedIn integration!

Finally, in order to protect the innocent, the screenshot above is test data generated by Benjamin Curtis’ Faker gem rather than our real contacts!

Although (at least with the bitnami stack) you can deploy your WAR files to Tomcat using the manager, sometimes you need to upload other files (e.g. database scripts) to your AWS instance.

I like using Filezilla as it makes FTP easy, so was happy to find I could use it to move local files to my EC2 instance.

It needs a bit of set up in order to use your AWS key but it’s pretty simple. Here’s a quick how to. You’ll need the following:

• Putty – You’ll probably be already using this to connect to AWS – download page
• Pageant – An SSH authentication agent for Putty – download page
• Your AWS key pair file converted to a Putty PPK file. There are a few resources that descrive how to convert a PEM to a PPK file. Basically you do this using PuTTYGen from the same download location as above.

Once you’ve installed all of the above, launch Pageant (in Windows it lives in your system tray) and then add your PPK converted key-pair file.

Then launch Filezilla and make a new connection to your EC2 instance using its public name. You can find this through right-clicking on the instance and then choosing connect. A dialog box comes up with the connection details.

Make sure that you select SFTP – SSH File Transfer Protocol as per the image and have the right user name and then you’re done. Happy transferring!

In an earlier post, I talked about the importance of wireframes as a communication tool. As we’re working with Android, being able to quickly create wireframes for Android phones is a must. I’ve looked around for a good set of Android shapes for Visio and couldn’t find anything suitable, so I have created some that you can download.

The library includes:

• project layout, chrome and notes
• navigation controls
• feedback controls
• lists
• form elements
• gestures

I think this is a pretty complete set but please let me know if you feel it is missing anything that I should add.

To make it easier to see how to use them, I’ve also included an example set of wireframes (visio and pdf) that you can download. It shows an android version of our time recording application.

I’m writing a bot to place bets using Betfair’s API. The smarts about what bets to put on will be provided by bet-guru “Meffa”. The easier bit will be the robot. It’s just a “fun” project (disclosure: secret hope is that will be on a beach in Antigua next year :) but a good opportunity to try something other than Rails-style projects using JRuby.

Betfair provide a full API and more importantly really excellent documentation and a full-featured demo app. Other API providers (and I include ourselves) please note. The API comes in different flavours, but I’m using the Java one which makes web service calls through Apache Axis.

I’m going to chronicle the steps to build our “botfair” bot although I’ll leave out the secret sauce of Meffa’s algorithms otherwise that beach is going to get crowded.

The first step is making a simple API call using from the interactive JRuby prompt.

Ingredients

• A betfair account. Sign up here
• The betfair Java sample extracted in to a directory, e.g. c:\dev\betfair-java-sample
• JRuby

Logging in and checking markets

Once you’ve extracted the sample application. You’ll need to build it and the WSDL stubs.

We’ll be calling methods from the \betfair-java-sample\src\demo\APIDemo.java class so it’s worth opening up this source file to take a look.

Build the sample and stubs
Before we can do anything we need to compile the classes and regenerate the web services stubs. You’ll need Java 5 for this.

Navigate to the directory and then run ant to build the class files and ant stubs to regenerate the WSDLs.

Set the classpath
You’ll need to make sure that all of the JARs needed by the sample are on your classpath. There are a lot, so I wrote a Ruby script to generate the classpath statement. Run it from the lib file to create you a batch file to do it:

jars=Dir.glob("*.jar")
f=File.new("set-classpath.bat","w")
jars.each do |j|
	f.puts("set CLASSPATH=%CLASSPATH%;c:\\dev\\betfair-java-sample\\lib\\#{j}")
end
f.close

Login and look at markets
In this part we login to Betfair from the interactive JRuby prompt, jirb. You’ll need your Betfair username and password.

1. Start a command prompt in the compiled classes directory (where Ant created your classes).

2. Set the classpath by running the file we generated above. This will make sure that all of the JARs are available (if anyone has a better way of doing this then please let me know!)

3. Type jirb to create a JRuby interactive session and then run each of the commands below separately in the session

require "java"
include_class 'demo.util.APIContext'
include_class 'demo.handler.GlobalAPI'
context=APIContext.new
GlobalAPI.login(context,'your_username','your_password')
types = GlobalAPI.getActiveEventTypes(context)
types.each do |e|
puts e.name
end

What did we do?
The result of all that should have been a list of markets that you can view or bet on using other features of the API.

This isn’t particularly useful as a bot to bet with but does act as a nice proof-of-concept to show that it’s pretty easy to rig up a betfair client using JRuby and Betfair’s API.

Next time …

The next stage will be to poll a particular market (will pick English football whilst the season is still going) and report back on potential opportunities.

Couple of weeks ago, one of my task was to code complex UI rendering logic which had dependency on the database values, there was an Object Relational Mapping(ORM) like Api provided without any source and limited java documentation. Being new to the client organization, there was a challenge of delivering the tasks on time and still provide quality code. Most of the complex applications have dependencies on the external services, legacy code/systems, database and of course complex build processes and dependencies,my project is one of those. Also you don’t have the luxury of getting your code to deploy and test at your will, and then if you found issues, debugging the code becomes painful with such dependencies, that is when mocking frameworks come to rescue.

So I decided to write a junit testcase and mock the ORM api objects. After doing quick googling over internet, I found mockito is the happening mock framework these days, so I dived in to Mockito, it took about two hours for me go through and test all the samples provided using eclipse and junit, may be I am slow, but it was more gripping I kept going through each one by one.

Of all the features in Mockito, I found argument matchers, callback stubbing and spy interesting. Together they make a good combination, in fact you will realize the importance of spy objects for complex testing, even though Mockito API suggests to get away with simple stubbing, I tried to exploit it, and works fantastic!.

Let us assume, class A which has some core logic and is invoking methods on class B,

• Class A —>> pre-processing + ClassB.method() + postProcessing;

Usually this is the case with clients invoking some services, the service being local/remote/webservice etc, and the post processing logic will depend on the response from the server.

Your focus is to test the client side logic in Class A and mock Class B. I have come up with a simple test case and some client/server classes and to demonstrate the interesting features of Mockito.

The class diagram below is the use case for the above scenario.

below is the listing of test cases, you could download the source code from our side

Client code

public int invokeComplexService(String operation ,int val1, int val2){
	ServiceRequest request = prepareRequest( operation ,val1, val2);
	ServiceResponse response = service.evaluate(request);
	int result = processResponse(response);
	return result;
}

Initialize the following in setup

//create a client
client = new Client();
//create a spy reference to RemoteService
spyService = spy(new RemoteService());
//for simplicity , let us inject the service reference here
client.setService(spyService);

Test service using spy(original methods)

//there is no mocking here, using spy to invoke the original methods
assertTrue(client.invokeComplexService("SUM",1,2) == 3);
//here, we should expect RuntimeException, as the operation is not supported by the service
assertTrue(client.invokeComplexService("SUBTRACT",1,2) == -1);

Test service using spy(mocked method)

//mock a specific method which is indirectly invoked by the client
doReturn(5).when(spyService).evaluate(anyInt(),anyInt(),anyString());
//now , this is the thing I would like to highlight the benefit of mocking
//here, there is lot of translation logic in client code thats needs to be tested,
//without making service call over the wire
//1. pre-processing of the service request 
//2. invoking the service 
//3. post processing the service response
assertTrue(client.invokeComplexService("SUM",1,2) == 5);
//Note: the we are asserting false here, as the evaluate will return 5 for all operations.
assertFalse(client.invokeComplexService("SUBTRACT",1,2) == -1);

Test service using call back stubbing

//mock a specific method which is indirectly invoked by the client
//interesting aspect here is that we could play with service input and return response we like
//rather than mocking a constant response, we can make the response dynamic based on the inputs
doAnswer(new Answer<Integer>() {
		public Integer answer(InvocationOnMock invocation) throws Throwable {
			  Object[] args = invocation.getArguments();
			  //process the response to handle any operation, SUM,SUBTRACT,MULTIPLY
			  int result = Calculator.doOperation(((Integer)args[0]).intValue(),
					  ((Integer)args[1]).intValue(),(String)args[2]);
			  return new Integer(result);
			}
	  }).when(spyService).evaluate(anyInt(),anyInt(),anyString());

//if you realized, testComplexService2 makes the same call, we get RuntimeException
//but here we are testing the client for the future operations/values that we could expect
//so, all the assertions below will pass
assertTrue(client.invokeComplexService("SUBTRACT",1,2) == -1);
assertTrue(client.invokeComplexService("SUM",1,2) == 3);
assertTrue(client.invokeComplexService("MULTIPLY",1,2) == 2);
assertTrue(client.invokeComplexService("DIVIDE",1,2) == 0);

I have used the following versions: junit 4.8.2,mockito 1.8.5, jdk 1.5.0_22. You should be able to run the test case after downloading the source , there more testcases in the source other than described above

Hope you will enjoy testing with mockito.

We are currently working with a mobile telecoms platform provider to help customise their point of sale product for use by an Albanian mobile operator. It’s a complex project and has a large delivery team and as such good communication is a key factor in our success.

The team is approximately 30 people spread over London, Ukraine and India. The connectivity in some locations is poor, and couple that with no common first language, communication is often problematic using skype or the phone. The requirements documentation has posed their own challenges with document authors being unavailable having moved to a new project or sometimes left the company altogether.

In the early days, the result was that the team delivered a solution which was not what the customer wanted, not what the business analyst expected but which took a lot of hard work from developers leaving them somewhat demoralised. We realised pretty quickly that this wasn’t workable.

Our solution has been to use wireframes as our primary communication tool. They accurately capture the flow through the system, the expected input to the system and response to the user and are language agnostic. They have made a huge difference to how close to the customer’s intended outcome we are and everyone feels more empowered and enthusiastic as a result.

We’ve iterated a few times to reach an ideal set and to save you time I have listed them below:

• provide a high-level screenflow which shows navigation pathways through each screen in the wireframe set
• label each screen clearly, give it a title and author with contact details and make sure that is shown when they are printed
• publish them in a format which everyone can open (and which doesn’t get trapped by virus checkers) e.g. pdf instead of visio/html
• include annotations to explain functionality and show functionality over a series of pages in your wireframes e.g. an autocomplete before any text is types, offering options, selected option
• if there are subtleties e.g. alignment is deliberate, treatments are deliberate (for mandatories), make these explicit either in a separate set of UI guidelines or using annotations

I have also captured some comments from the developers and the client below, just to illustrate how it helps everyone.

The last post a good month and a half ago spoke about porting our revenue management product Jair to JRuby and deploying on Amazon’s Elastic Beanstalk.

Well, we’ve ported the app and apart from a few hiccups with gem versions not being supported in JRuby – we had to upgrade from the CSV to FasterCSV gem which I guess is no bad thing! – all went smoothly.

I really like JRuby as there’s something satisfying about writing in Ruby but deploying on to Tomcat. Even more satisfying is when the application actually runs up on Tomcat!

I’m not yet fully convinced as on shutdown Tomcat issues dire warnings about memory leaks being “likely to happen” but we’re going to do some application profiling soon so we’ll see.

Rather than use Elastic Beanstalk (only available in Amazon’s US regions at the moment) we just deployed on to a customised Bitnami Tomcat stack on AWS EC2. Again, this seemed a lot easier than the last time I tried although perhaps that’s down to having the right tools installed beforehand(!).

I’ve got a bunch of tips / things that we learned from the porting exercise and also AWS deployment so I’ll post those up in the next few days.

For us, AWS rather than GAEJ, is the way forward. Although there are different services on GAEJ (can only think of caching and IM offhand) so it could be useful for some apps, but how long before AWS offer the same? As well as that with AWS there is the freedom to install things like memcached etc.

I do feel a bit guilty though every day at 4p.m. when the True North chatbot (one of our apps on the Google App Engine) pings me on IM … maybe it’ll be its turn for porting next.

I was at a webinar last night about AWS Elastic Beanstalk, Amazon’s newly released web app cloud deployment platform.

The timing of the webinar (10 a.m. PST == dinner time in the UK) meant that I had to leave before the end, but it was a pretty interesting introduction and left me wanting to check out.

AWS Elastic Beanstalk is a Java-Tomcat stack that allows easy deployment of JEE applications (via AWS itself, via an Eclipse toolkit, via the command line, or SDK/API). Perhaps the best thing about deployment is that you can change a bunch of configuration items a lot easier than having to redeploy.

The usual but incredibly useful stuff about scalability and growth being taken care for you holds just as it does for Google App Engine for Java. As does the fact that there’s a console with which you can monitor your apps – again via the AWS dashboard.

It comes with database support (MySQL, SQL*Server, Oracle, DB2). JRuby support is also in there and the stack seems more standard that the GAEJ stack. That coupled with the fact that you have root access so can go low-level with root access means a lot more flexibility than GAEJ gives.

AWS have a sample app that can be deployed so you can see although I’m not sure I get the point of that really (won’t anybody inclined to or capable of deploying will probably have their own sample Java web apps they’d like to try out anyway?).

Other platforms are planned, but Amazon were coy about even mentioning let alone committing to a roadmap on the webinar. I figured they just hadn’t decided yet but that’s no matter.

We’re planning to try it out in the next month or so with a JRuby port of Jair so will post up our findings as we go.

I found this tedtalks.com presentation by David McCandless really enjoyable and enlightening. I particularly liked the way he had mined data from unstructured sources and then brought it to life visually. There were also some wonderful choices David had made to make his visualisations more usable:

• containment – David showed OPEC’s climate change fund contained inside it’s revenue
• categorisation – David showed types of spend in the Billion Dollar-o-gram to show type and also in the fears chart
• making figures relative – David re-cast figures of spend on military budget against GDP and allowed a clearer comparison
• comparison – David gave charts something to compare against e.g. UK budget deficit where the user might have needed something that made more sense to them against which to compare.

Do take the time to watch it. You’ll feel inspired!

Sometimes it can be confusing as to why your expression-language expressions are not being evaluated in your JSP.

The confusion is almost always because of misunderstanding of when EL expressions will be ignored by default and when they will be evaluated.

EL evaluation is controlled in four places:

1. The servlet version your web server supports
2. The servlet version specified in your web.xml file
3. The el-ignored element in your web.xml file
4. The isELignored page directive in your JSP

Servlet version

Your container must support the version you specify in your web.xml file. Don’t try and go higher if your web server doesn’t support it, it may not drop down cleanly to the highest version it does support.

Consult the documentation for your web server, for Tomcat:

• 5.5 – servlet 2.4
• 6.X – servlet 2.5
• 7.0 – servlet 3.0

Servlet version in web.xml

Include the appropriate version and namespaces in your web-app element.

<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="http://java.sun.com/xml/ns/javaee"
	xmlns:web="http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
	xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
	version="2.4">

el-ignored element

In your web.xml file, set the el-ignored element. This setting will apply right across your web-app.

<jsp-property-group>
  ...
  <el-ignored>false</el-ignored>
  ...
</jsp-property-group>

isELignored page directive

Finally, you can set whether your expressions will be evaluated in your JSP. Set it to false and expressions are evaluated

<%@ page isELIgnored="true" %>

This (unimaginatively titled) post explains how to set up remote debugging for Tomcat6 running as a Windows service using Eclipse and first, why you might want to.

Why remote debug?

Once an application grows over a certain size it becomes more difficult to work out what’s going on from watching its behaviour – e.g. error messages and stack traces. This is equally true for small but layered or componentised applications (and doubly true for large layered and componentised applications!).

One approach is to add log statements outputting debug information like variable names, loop counter statuses, null checks, but this quickly become more time-consuming than it’s worth, and being able to debug in the IDE gives you more control and ultimately saves a lot of time.

You might not believe me, but try a few rounds of entering log statements and restarting and then we’ll talk. If you’re not even doing that and entering System.out.println then shame on you!

How to set it up for Tomcat6/Eclipse

This article covers setting it up for Tomcat6 only because it’s slightly different. There are undoubtedly resources out there for your favourite web server / IDE combination so take a look.

In order to remotely debug the IDE needs to connect to another port where your app server is running (another meaning one than the port on which it accepts HTTP connections).

There are two parts to making this happen:

1. Telling the JVM on the web server which port to use for remote debugging
2. Telling your IDE which host and port to listen to

Tomcat6 is usually set up as a service on Windows and that makes setting the JVM arguments a little more complicated.

1. Open a command prompt in your /bin folder, e.g. “C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin”
2. Use the tomcat6.exe utility to update the existing Tomcat6 service to add the JVM arguments. The arguments begin with -X and are separated by semi-colons.
   

   tomcat6 //US// —JvmOptions -Xdebug; -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n

The first argument tells the JVM to remotely debug and the second to accept a socket connection on port 8000. You don’t have to choose this port, any sensible free port will do.

That’s the web server set up. Now you need to set up Eclipse.

1. Go to Run>Debug Configurations and click the icon at the top of that screen to create a new one
2. Enter the host (in the picture I’m running my server locally hence localhost) and the port that you configured it on.

That’s all the set up. Now you just need to start the web server, set a breakpoint (e.g. in your login.jsp) and navigate to the login page on your development server. You should see your breakpoint hit.

Good luck!

We’ve been working closely with two charities recently, so when I was reading through Jakob Nielsen’s alertbox archives and saw his article on Donation Usability it caught my eye.

The article describes the steps non-profit organisations should take in order maximise donations. In short it’s descriptions of two things:

• what the charity stands for (and in particular mission, goals and objectives)
• what your money is used for

People want to know what the charity stands for so they know it shares their ideals and values, then knowing that, how will they do it.

To get this right, the following things are important:

• answer the questions at a high-level on the home page
• have a clear IA to allow the user to find the underlying detail
• make the entry point to the donations process clear
• make the donations process simple (although this is usually a third-party service)

Another useful tidbit was that websites were more for small-scale impulsive giving but newsletters are better for long-term donor loyalty.

This was from the short report but there is a whopping 124-page report for the people who have got the simple stuff right already.

I’m doing some work at a client’s where I needed access to a drawing tool and didn’t have a PC with Visio on it.

I’ve searched for online drawing tools before without much success but optimistically thought I’d try again. Reasonably near the top of the list was Gliffy which I thought I’d give a try.

So far (and it’s only been a few hours) I’m really pleased with it as it’s really easy to use and has done reasonable ER and UML sequence diagrams. I actually think it’s more pleasant to use than Visio (not difficult) but admittedly doesn’t have a lot of its features.

It’s not free but the single user licence is $5 a month, so not too expensive.

I reckon it’s a viable alternative to Visio for general diagramming (you couldn’t use for wireframes) and probably worth it for ease of use and portability. I’m going to try using it for all diagrams over the next month to see whether the theory pans out.

Last week we were adding features to a special version of Falcao ready for Richie’s World Cup prediction competition.

It’s been sponsored by Academy HR which also means there’s a prize for winning.

We’ll be adding features up to and after the first game, but if you’re interested in playing then sign-up here.

A bit of a specialist post, but I was caught out today by odd behaviour with the using the memory_store cache in Rails.

This is a code snippet from our controller method

cache_key = 'current_standings'
@standings = Rails.cache.read(cache_key)
# check cache
if @standings.nil?
 # ... fetch the data
 # add to the cache
 Rails.cache.write(cache_key,@standings)
end

The problem I found was that the second time through, i.e. on the cache read, the items in the cache were nil.

When I tested the items in the cache there were the number expected, and they were the right type but they were just not populated. Putting the object in and out of the cache in the console was fine.

In the end, and after a lot of error and some trial, I found that I needed to set the config.cache_classes property to true in the development.rb file. An unfortunate side-effect of this is that hot deploy is no longer hot and I need to restart the server to pick up changes but at least the cache works okay.

I’m guessing that with each request the cache (or at least the variables in there were being blasted) but would be interested in hearing from anyone else who has experienced the same/similar.

This demo of a binaural recording is pretty freaky.

If you close your eyes it’s almost exactly like being at the barbers (always assuming your barbers have in-house guitarists and slightly dubious Italian accents).

Thanks to actor and sound guru David Charles for the file and 5 minute explanation of binaural recording on the train to Clapham Junction.

To get the best effect of the recording listen to it through headphones (and close your eyes :-)

In my last post I was evaluating Infobright’s column-based database (ICE) against some data in a warehouse we’ve built for one of our clients. We compared storage space and timings and results were pretty impressive and I thought I’d share.

Versus an unoptimised MySQL InnoDB database ICE was between 15 and 40 times faster on the four analytical queries we selected. In addition, ICE used over 10 times less storage space.

We also tested ICE against a database that was optimised heavily in favour of the queries we were running. This test was pretty unrealistic as no warehouse I’ve ever worked on has fit all of the queries that run against it. Despite this, ICE was 2 – 6 times faster than the optimised database and around 17 times smaller.

The test case we used only had 1.3 million rows but the results were impressive nonetheless. Probably the best thing about it was how easy it was to get it up and running and performing to that level – compare that with the brainpower and effort needed to optimise a database. I know this was a limited test but it was still on real data and asking real business questions.

Anyway, we’ve put up the full report and approach on Scribd so take a read if you’re interested and either comment here or get back to me for more information.

There are a lot of good user interface pattern libraries that are available now – UI Patterns, Welie etc. which covering both general and specific topics.

Although patterns are often categorised e.g. Search, Navigation one thing I haven’t ever seen, although maybe it exists, is a set of macro patterns.

I think there is a lot of value in having some coarse grained patterns that you apply to a specific scenario that as you consult with the client you remove/include elements of.

If we take the example of an Inbox. Inboxes are useful in any situation where you have a pool of resource to allocate work to. Depending on the way your pool of resource operates and is managed determines how you might apply the pattern.

The basics will be there:

• sortable columns
• sensible default sort
• filtering
• icon columns for space saving and clear communication
• highlights e.g. new or unread
  etc.
• actions
• selection approach

Then there will be some variable elements

• allocation statistics
• filtering
• rules
• pagination

and many more, all of which depend on your need.

We will be releasing a set of macro patterns, adding to them as we complete client projects. If you are interested in hearing more or speaking to us about them, please contact me

Last month I went to an interesting seminar given by MySQL where one of the presentations was by Infobright. It’s an open source analytics solution and their sales guy was plausible enough to get me download it.

I’m currently using it to do some analytics/Business Intelligence testing running it against some historical data from a data warehouse we’ve built. I’m asking it things that I know relational databases struggle with, or at least require a lot of tuning input to get working nicely (e.g. top-n queries to show the most valuable customers, etc.)

At the moment the performance improvements for are pretty significant (15x quicker) with only a month’s worth of data. I’m hoping that once we load a quarter’s-worth, or a year, or multi-year then the improvements will ramp up even more.

The nice thing about it is that it’s built on MySQL so you can run regular SQL against it and it does all the clever stuff.

The weird thing about it from a relational viewpoint is that there are no indexes or anything like that. It’s a column-oriented database and hence great for group and aggregate functions – the mainstay of analytics.

Once I’ve done a bit more formal testing I’ll publish out the queries we used along with the comparison to the traditional relational database we have currently implemented on.

Quick update:
Just finished the first round of testing with around 625,000 rows of data which represents about 1/3 of a year. Infobright was faster on all the queries ranging from 15x to 30x faster.

We’ve just finished a presentation on tips and guidelines we’ve learned whilst working with the GAEJ.

It covers issues problems detaching objects, embedding associations, when to denormalise, and gives some advice on data modelling and when GAEJ may or may not be a good choice.

Take a look and we’d be really interested to hear your tips and experiences too.

We often work with clients on their internal systems and in all the time of doing that, I’ve yet to see a content management system which allowed business content creators and editors to work without any IT involvement and create sites that were:

• consistent with the rest of the user experience (other than by being bad)
• accessible

It seems that over time an HTML editor role has crept in and now it is not just accepted but expected. In the worse cases when the business content editor wants to make a change they have to create change requests which go through a development change process with all the extra time and cost bloat that that includes – madness!

I think it happens for a number of reasons:

• a low skill level of the developer when creating the system HTML templates
• limited flexibility in those templates, meaning that the HTML content editor layers lots of complex HTML to achieve their goals
• horrible WYSIWYG editors that allow content editors to create inaccessible pages
• no training/support for business content editors so they don’t know what makes a readable, usable, accessible page

Poor HTML skills and missing requirements about template types are simple to fix and no one should be in that position.

The content editor is more difficult though. The temptation is to buy an off the shelf editor thinking it will be easier for the business to use and therefore it’s job done. The reality is that it’s quickly out of the hands of the business because sooner of later there needs some custom HTML to do something that the templates don’t support. The markup they create often has accessibility issues too – “features” in the editor which create output fixed font sizes, headings out of order, poorly embedded flash etc. The .Net controls seem to be the main offenders. My feeling there is that wiki markup for content areas within templates is still a good solution and quite simple to follow provided there is a guide available right there in the editor page.

I’ll post some wireframes shortly to illustrate what I feel makes a simple yet effective CMS back-end.

I came across this Google Wave Extension Wish last week and thought its spirit matched that of one of our 2008 posts about social workflow.

We tried out Wave, but I’m not really convinced that it’s any more suitable for workflow than traditional email (the state transition is only part of workflow, just as important are the dashboard and metrics elements).

I still do think there’s a need for a simpler workflow tool that let’s human users decide when something is complete or complete enough.

We’re currently developing a simple workflow component that’s intended to be easily dropped in to products where there’s a need for workflow or task management.

We’re reviewing our design with a selected audience, but once that’s complete I’ll post up our functional architecture and the principles around open workflow for reference and discussion.

If you’d like to join the initial review process it’s not too late … just email me to discuss.

Google announced a new version of the App Engine SDK today with a couple of interesting features.

Given the strength of feeling in evidence in the forums and blogs I expect it’ll be the raising of the 1000 limit on datastore queries that will get the most comment, but I was much more interested in the unit testing framework.

Unit testing support in the GAEJ has been non-trivial, which I suspect pretty much leads to non-existent testing in most apps. If the support makes this easier – especially in testing datastore access and caching – then it’ll be a hit with me.

There already was a way of simulating the app engine through a test classes supplied by Google but I’m hoping this will make things easier.

We’re going to upgrade the SDK and test out the new test framework and will post our results here.

We’re fans of Watir which amongst other things we use to automate web browser tests.

It makes automation very easy and you can do a lot with nearly no coding. What coding you do have to do is fairly intuitive.

However, looking at Sikuli it makes Watir look very complicated.

I don’t think it’s going to replace testing, as it’s not obvious that there’s a way of testing the resulting page (unless you can take a screenshot of the whole page?!). Anyway, once I’ve downloaded and tried I’ll post here with results. It seems too good to be true, but we’ll see!

Thanks to Ed who brought this to our attention.

We had a table-top version of Gorf at our student house in Canterbury, way back in 1987. Its arrival had been heralded for weeks, and when my housemate Mark finally brought it through the door – with help as these things were heavy – we did little else but play it for the whole of the winter term.

Looking at the video, you’ll probably laugh at how primitive the game is – my children did anyway. But Gorf was cutting edge when it was released, and even in 1987 it was still unusual in that it was one of the few games to “talk”. The voicebox in our battered version was very inconsistent, which only gave it an added piquancy as every now and then it would bellow unintelligible robotic pronouncements and then fall back to silence. A bit like some people I’ve worked with really.

Now, not only does Gorf look terribly dated (even I didn’t watch all 4’29" of the video) but I bet it’s really easy! There were very few buttons to have to remember, but at the time it still took us weeks to master. The thing is no one minds that now. It was a perfectly good game at the time and I bet most people who played it look back on it fondly.

It’s the same with application design – sometimes we get a bit too caught up on designing for the future when actually it’s plenty difficult to design something good for today. I’m the first to admit my shared guilt in this as there have been times in my career where I’ve over-designed for re-use.

Re-use is no bad thing – we rely a lot on reusable components when building our applications – but it shouldn’t become an end in itself. The important thing is for users to be able to use your application as soon as possible, and it might be worth sacrificing some design options to achieve this. Better that than it lying for months / years in development whilst you try to make it fully generic. If it has a longer shelf life because of some of the re-use features you put in then that’s a great bonus – but you should think of it like that – as a bonus.

Why does over-design happen? I think it’s a combination of a couple of factors.

Software developers that grow in to designers want to generalise things. It’s more pleasing to think that not only does your application make a good e-commerce tool, but with a bit of configuration it could also be a perfectly serviceable missile-tracking system.

The other consideration is that, as a designer, you’ve probably spent the last couple of months prior to development convincing your manager that you need to build a framework as it will save development costs later. Designers love to code nearly as much as project managers love to be seen saving money. Again, I’ve nothing against frameworks – we have a few ourselves – but these days I think they should be developed after you write the first two or three apps rather than before.

Software obsolescence is inevitable – even for classics like Gorf. It’d be a shame for software you’re designing to become obsolete whilst you develop your framework before anyone has had a chance to use it.

Finally, a plug for my favourite London bar, Fluid in Smithfield where you can play on working table-top and stand-up 1980s vidoegames (sadly not Gorf) and as if that weren’t enough, it’s a great chilled-out venue to enjoy a few drinks.

Last year I did a contract for a startup consultancy in which I was the only person with IT application know-how, everyone else had plenty of business know-how. This created an interesting situation where the excitement of a startup winning its first client threatened to swamp any IT strategy. The eagerness of the team to see results immediately was a daunting prospect as the previous project I worked on had quarterly releases and teams for dev, testing and deployment! So I had to find a different approach, that’s when I tried a web app framework called Hobo which is a set of plugins on top of Rails.

Hobo does clarify one deviation from RoR regarding Interface First approach. The development starts with sketching the Models rather than designing the interface (“what people see is what you’re selling”), this is explained by an ambitious statement about rewriting the rule because “…Hobo, is actually quicker than creating html designs.”

Hmmm, lets run with it a bit longer….

I created the app, with the hobo command, which looked almost identical to a standard RoR project. So then using a Hobo generator to create the model resources (much like the Rails scaffold generator) gives me controllers and models as you would expect but there are no view files. This is very much like Groovy on Grails where the (default) views (and routes) are generated at runtime and you only override if you want to change them. The Controllers were much leaner, containing just 2 directives to the Hobo framework for the default behaviour and conversely the Models had the field definitions and some Hobo permission methods for access control, so more than usual.

And where were the DB migrations? Well these are created by another Hobo generator which is quite clever. It looks at the Model code and compares it to the DB schema and creates a migration file to get the DB from current state to the new state as defined in the Model. This works for adding, modifying and removing fields in the Model. This turned out to be a life saver as it drastically reduced the turnaround of Model changes so that they could be done during a demo to incorporate some feedback immediately or show other possibilities. Ah that’s fine for flat table structures but what about relationships I hear you say, well it handles those too and even adds indexes where necessary. This allowed me to try out new things quickly in front of the team, safe in the knowledge that I could roll back the code and DB to a working baseline in a couple of clicks (in Netbeans). I liked this feature a lot.

Hobo gives you a lot by default which you can remove or modify very easily. The first to strike me was the default user model created with each Hobo project which allows signup and basic admin such as editing or resending a forgotten password. This together with the permissions I mentioned earlier, specify who can do what to each Model. Also, in dev mode, Hobo displays a “user changer” which allows you to navigate the app and see it from different roles (admin, user and guest roles available by default). You can also specify permissions to maintain data integrity and the UI will reflect the change. E.g. by default you can change the parent of a Model in the UI. If isn’t what is required you can withdraw that permission and the UI will automatically not show the drop down in the edit screen. So when I was asked to make a subset of data/functionality available to certain group of users by the next day, it was possible with Hobo not least, due to this feature.

Another default that is neat is when you want the view of a Model to list its children, it is a simple directive in the view hints file for that Model e.g. children :tasks. What this also does is when viewing a Model that is a child, the parent is shown as a breadcrumb navigation.

Other default behaviour includes pagination and an application wide search facility.

So a little work on the models and controllers and the UI was enough for the team to try out and give feedback and refine the data model … in a matter of days. It was as close to the 80-20 rule as I have seen.

Of course we needed a little more than the default UI but that will wait for another day.

This is a TEDtalks from 2006 which I only found over the weekend.

I found it really enjoyable and there are lots of useful learnings for user interface / application design. Barry talks about the implications of a growing number of choices (giving us freedom) and where responsibility for making those choices now sits. He goes on to talk about how the number of choices impacts your level of satisfaction level after having made a choice.

It definitely made me think about how you deliver choices to users in applications, how many to give on a given page both to maximise their ability to make a choice (avoiding the paralysis of having too many) and also their feeling of satisfaction after having made it. For me this means your application has to know a lot about the user and then use that to serve up a limit number of useful choices to support them in achieving their goal. Easy peasy!

Lets see what difference that makes to True North applications in future.

It’s quite common to need to set an attribute value at runtime for a web application. Titles are a very common e.g. using a person’s name as the title for their profile page.

We’re currently using tiles for layout management in a GAE4J application so as it took me a little while to decipher the documentation, I thought I’d just write it up quickly here.

The steps are pretty simple:

• Create an implementation of ViewPreparer
• Link your ViewPreparer to your tiles definition.

A ViewPreparer is called before a definition is rendered.

Creating a view preparer

You only need to implement the execute method. The execute method gives you access to the TilesRequestContext which you can use to access request or session variables. They are where I would expect you to be pulling your dynamic attribute value from. An example is shown below

public void execute(TilesRequestContext tilesContext, 
    	AttributeContext attributeContext) throws PreparerException {
    	// get the session attributes
    	Map<String, Object> sessionScope = tilesContext.getSessionScope();
    	
    	Survey survey = (Survey) sessionScope.get(BaseServlet.SURVEY_ATTRIBUTE);
    	if (survey != null) {
    		String title = survey.getName();
    		
    		// if a name has been set, show it as the title
    		if (title != null && !title.equals("")) {
    			attributeContext.putAttribute(
    				"title",
    		        new Attribute(title));
    		}
    	}
    }

Linking the ViewPreparer to your tiles definition

Set the preparer attribute on your tiles definition. See the example below.

<definition name="survey" extends="admin" preparer="com.ontruenorth.socrates.view.preparers.SurveyTitlePreparer">
	...
</definition>

Enjoy!

Don’t worry, we’ve got the fire brigade on speed dial if you get your wheelchair stuck.

Well spotted Leonie.

This is a nice example of a paper prototyping session for a mobile interface.

I was looking through our blog the other day looking for an old post which had something in I wanted. I thought I would speed up finding the post by using the tag cloud and it struck me what a poor interface control they really are. I think that it comes down to a number of issues:

• context
• volume
• order
• underlying issues with tagging in general

Context

I don’t remember the last time I saw context information for a tag cloud beyond the page it was associated with. Typically you might find them on a blog or a page of case studies so you can guess that maybe these are all possible tags but it’s rare to see any kind of time information as a label e.g. Tags for last month’s articles

Relevance

To support the context point above, I also don’t always want all tags, I want ones that will help me find what I’m looking for and show me trends. I may have written a lot about West Ham at the start of the year but be too embarassed to talk about them now but my shame will be revealed by a cloud showing tags for the whole year. Perhaps it would be better to show two breakdowns, the last month or three months (obviously depending on what you’re tagging) and everything.

Order

There doesn’t seem to be an emerging convention across sites what order show tags in. Some show in the order you first used the tag, some show in alphabetical order, some show most used first. I think alphabetical order is probably the least useful as different people will use a different tag for the same thing – you say macbeth, I say the scottish play. Then first used and most used well again I think this depends on when/why you are displaying the tags.

Tags

Yep – I do think some problems just come with tagging’s baggage. It’s so easy for you to end up with a number of tags which are really should be the same e.g. socialmedia, socialnetworking (social media eskimos can argue amongst themselves that these are really different). It’s easy to end up with huge numbers of tags even with a small number of things to tag. I recently looked at a competitors case studies and there was a full web page of them – horrible

So, clearly if I’m writing this post, we couldn’t leave out pages using tag clouds. We’ve now switched to summary tables in usage order (clearly labelled of course) with a full list of tags should you need them. I’d be interested in whether anyone has ever done any usability testing with tag clouds and what they found.

Last night, with some time to kill in New Malden, the kids and I decided to pop in to a Korean supermarket. There’s a big Korean community in that part of London, and as well as delicious-smelling restaurants there are lots of minimarts that remind me of the Indian and Caribbean stores in South London I used to go to with my grandparents.

The staff were very friendly and regarded us with curiousity and amusement as did the exclusively Korean clientele, and it was only when browsing that I realised that all of the packaging and labels were in Korean (I know,I know). For a while it was really disorienting and we all ended up gravitating to things that we recognised – Korean pot noodle equivalents for my teenaged son, biscuits for my daughter, and packets of seaweed for me.

It made me realise how the visual components and behaviours that we take for granted when developing our applications could seem just as bewildering to users not familiar with them. It made me glad in turn that we have Mark and Lilamani’s usability expertise to hopefully reduce the sense of vertigo for our end users, and I made myself a promise that we would keep factoring in usability testing for all of our products before they see the light of day.

Anyway, I don’t want to labour the point. The kids and I ended up buying a couple of items, all of which were what we thought except my “savoury snack” Jolly Pong which ended up being a very tasty Korean version of Sugar Puffs. Looking at the picture on the packet (Korean writing over what look like Sugar Puffs) I don’t really have an excuse for my assumption other than it came in a savoury snack looking package What’s Korean for caveat emptor?

Finally, a plug for the shop itself. It was called kmart and they were very friendly and very reasonable, even before the 10% discount we got! We’ll definitely be back to stock up on the Jolly Pong and maybe some more exotic/healthy options.

The song needs no plug as far as I’m concerned, but for readers born this side of the 80s here is a live version on YouTube (sadly I couldn’t find a live performance).

It took me quite a bit of searching and reading to find why although Google App Engine was using JSP 2.1 that my expression language expressions weren’t being evaluated.

You need to add

<%@ page isELIgnored="false" %>

into your JSP and then you’re away

I was caught out the other day whilst writing a data service when I got a JDOException telling me I was accessing a field (in this case it was a serialized Collection) that hadn’t been detached.

This happens when you try to access an object, or collection within that object, which is still under the control of the PersistenceManager.

The PersistenceManager manages the persistence lifecycle of datastore objects, making sure they are synchronised with the underlying store. This is great in the data service layers, but most web applications will want to manipulate and display objects outside of this control before putting them back in to persistence-land. This is what detaching allows you to do.

The JDO spec is well worth a read as it describes the lifecycle in detail (a lot of detail :-)

There are several ways (including doing it manually) how you can handle the persisted object’s lifecycle and detaching in particular, but as we use transactions in our app the easiest was to use the JDO feature detachAllOnCommit which detaches an object and its collections. The code snippet below shows how.

Incidentally, although you can set detachAllOnCommit in JDO properties I prefer the explicit approach as it doesn’t hide potentially complex behaviour in configuration.

pm = jdoPersistenceHelper.getPersistenceManager();

pm.setDetachAllOnCommit(true);

tx = pm.currentTransaction();				

tx.begin();					

p = pm.getObjectById(Participant.class,participantId);				

tx.commit();					

pm.close();

...
// manipulate the paricipant here, for example
MailManager.invite(p.getEmail());

One final word about using Transactions in the Google App Engine. You can only enlist objects in a transaction that belong to the same EntityGroup. This means an object and its children (even objects of the same type are in different EntityGroups). We designed our application so that this would be easy for us, but it required careful thought and an app engine specific design. There’s more about transactions in the Google App Engine here which also addresses other ways to detach.

Recently, we’ve been spending a lot of time enhancing our existing products and deciding which of our new ones make it out of the lab and in to development.

In order to aid the latter we put together a set of seven principles. I don’t pretend these are exhaustive or anything more than common-sense, nor are they suitable or relevant for every organisation, but they’ve helped us formalise and speed our decision-making as to which products to focus on.

We’d be interested to hear your thoughts on this and whether you’ve come up with anything similar. As to our products, we’ll be putting a section up on the website soon to showcase existing and upcoming products.

I: A product must have a raison d’etre

• Who will buy or use this product?
• Why will they choose it over others?

II: A product should have a roadmap

• How will this product evolve?
• What truly makes it a product rather than a piece of software designed to solve a single / local problem?
• What is the bare minimum this product should do (i.e. what’s sufficient for a 1.0 release)?

III: A product must have a champion

• Who will set and re-set the business and technical direction of the product (we expect this to be two champions in reality)?

IV: Products should be designed to be flexible

• They should be easy to integrate.
• This allows products to be formed to make a suite (which increases the value of the products).
• Don’t reinvent the wheel and use existing standards where possible.
• Only specialise (i.e. use proprietary formats) where requirements dictate or it creates significant competitive advantage
• Favour open-source libraries.
• Don’t rely on clients’ data models (instead integrate with them and adapt to a neutral or
  standardised data model).

V: Products should use established and widely-known technologies

• Greater selection of development partners.
• Easier to sell to clients.
• Greater scrutiny of widely-known technologies means they are (usually) more robust than others.
• The exception should be true innovation where the development could not be done in existing technologies or it is significantly quicker to develop in newer technologies.

VI: Use technologies sparingly

• Select a few, established and robust technologies to build products with.
  • It’s easier to support more products with fewer people.
  • It’s easier to innovate (as there is a shallower learning curve).
• Technology choice and migration will need to be partly led by the market – i.e. if a new technology emerges and begins to become the leader (e.g. a new, popular language or platform) then we may need to port our products (e.g. moving products on to the Google App Engine)

VII: Build in diagnostics from the start

• Instrument the product to make support and troubleshooting easier

There is a surprising amount of discussion on this topic, some with convincing examples supporting one or other side of the fence. So I thought I’d have a go at explaining it…. Ruby uses call-by-value. There I’ve said it, now for the backup.

It hinges on the clarification between call-by-reference and call-by-value where the value is a reference which Ruby uses.

Call-by reference – means that the actual reference is passed to the method. This allows the reference (and/or the referenced object) to be manipulated in the method and those changes to be reflected once the method returns.

Call-by-value where the value is a reference – means a copy of the reference is passed to the method. The scope of this reference copy is just within the method. This allows the referred-to object to be manipulated via the reference copy and those changes would be reflected on return of the method BUT any changes made to the reference copy itself will not be reflected on return of the method.
E.g. assigning a new object to the reference copy would only have the scope of that method, i.e. on return the original reference would be unchanged.

I think this explains the behaviour in the examples I have seen.

We recently added a twitter feed to our homepage. It was a good little user experience lesson and took a few iterations to get to something useful.

Here is a quick summary of the decisions we made:

• We wanted to show something current to attract interest back to the site, so we chose to just show the last 5 tweets only
• As we already have a blog and showing all tweets at once meant they were pushed below the fold, so we chose to use a carousel
• We chose to show each tweet in the carousel for 10 seconds. We had initially used 5 seconds but that didn’t give the user enough time to read and to follow a link in the tweet vanished
• As some users may have already read earlier tweets or just want to scan them, we gave them the ability to skip forward and back
• To get the maximum possible value we made all elements of the tweet clickable – links, users and soon we’ll add hash tags. We also made them open a new window rather than leave the site, so the user can open each link to read and stay on the site to open the next

They were our thoughts – what are yours?

This announcement from Phusion on Ruby Enterprise Edition (REE) was posted this week, and it’ll be interesting to see the uptake and also what other REE offerings appear in the next few months.

We’ve no plans on doing anything with REE at the moment as we go at the pace of our Ruby hosts but we’d would be interested to hear from anyone who is using it or plans to use it.

A few years ago, I was at a presentation by Don Norman at the NNGroup Usability week. During his presentation, Don said that “the product is more than the product”. It didn’t really hit home at the time quite how much extra product a product can really have until recently.

We use Ocado for our weekly shop and the user experience is fantastic, more than just online shopping or food delivery, everything is set up to make it as simple as possible for you.

Your shop is delivered in a specific, short time window. Contrast that to other delivery companies who give you a half day or worse still a day window.

The printed bill that the people doing the delivery give to you is organised into what you will do with your shipping next – cupboard, fridge, freezer. Better still the food is organised into bags in the same way and they are colour coded.

The bill also summarises where you have saved money all into one section. You could say this is good marketing for them but it’s also good in financially tight times to see how much you’ve saved.

Finally, they’ll pick up your bags from last week for recycling and give you a free paper. Genius!

All of those things are separate from the online shopping experience which is equally as pleasing and now available on the iPhone

Well done Ocado. Showing everyone how it should be done!

I recently bought a new PC and have been installing a bright sparkly new set of 64bit applications and marvelling at the speed of them and highlighting the lack of speed of my brain at the same time.

I was just going through the process of checking out our rails apps and getting them up and running when I hit on issues with rails connecting to 64bit mysql using the mysql gem.

The error it showed was:

193: %1 is not a valid Win32 application

After trying a lot of different solutions, this one finally worked. Wonderful and simple, courtesy of stackoverflow.com and osdir.com

Simply copy this libmysql.dll file into your ruby/bin directory and hey presto!

I was happy to see this blog posting which announces incoming email support for the App Engine. It definitely makes it possible to create more powerful applications with better legacy integration. We’ll be trialling out shortly and post here with our findings.

Today I saw this story about injunctions being served on Twitter and as well as making me smile it got me thinking about the perennial problem of identity on the internet.

It’s increasingly difficult to assert your identity or ensure that the person or corporation you are communicating with is who they say they are. This is made even more difficult through the availability on the internet of photographs that can be uploaded as persuasive profile shots. Identity and authenticity is relatively easy to assert within a company (and also people seldom pretend to be others) as the company provides you with your identity in your email / staff number / directory entry etc. However it’s a different matter in the wild.

Twitter have taken a step of putting an ‘authenticity seal’ on their pages in short, acting as a trusted third party which is a pretty progressive (and sensible) step, and it got me wondering about how soon other social networking sites will need to do something similar and the sheer logistical effort needed to do it properly (even assuming it could be done properly) and the type, brand perception, and size of organisation you’d need for it to be truly trusted.

It’d be interesting to know as well how many of the people following the spoofers are aware of the fact it’s not the real person and what in reality will be able to be done in the future and outside the corporate domain. Incidentally, I didn’t know that this was the origin of the real McCoy … always assuming you can trust Wikipedia that is.

As many of you already know, on 5th September, Andy, Rich and I are running the Jungfrau marathon in Switzerland in the aid of the charity REACT (http://www.smartgive.co.uk/funds/jungfrau/). I’m writing to ask that give a little of your hard-earned cash to sponsor us and make the weeks of us walking like cowboys following the event seem worthwhile.

The Jungfrau is billed as being “the most beautiful marathon in the world”, so with visions of strolling effortlessly through sun-coated green hills we all signed up but were cut right back down to size by the small print …

• It is indeed 26.2 miles long, however almost all of it is up hill (up a mountain to be precise)

• At the end of the race our capacity to breath will be a fifth less due to the effect of altitude .. always good as I’m sure we’ll have lots to spare by then

• Most people (well those that know what they are doing) view the Jungfrau as the pinnacle of their marathon career – not the start

Because we can’t afford to take the summer off and train at altitude in Switzerland unlike the other 4500 competitors, we’ve developed a sophisticated regime to simulate exact race conditions and allow us to mentally prepare. It involves the purchase of a cuckoo clock and eating large quantities of Toblerone. We have found there is a limit to the amount of Toblerone one can face eating and we haven’t slept particularly well since the cuckoo clock arrived. Right now the Jungfrau is looking like a walk in the park!

The charity is REACT which stands for Rapid Effective Assistance for Children with Potentially Terminal illness and is a dynamic charity working to improve the quality of life for financially disadvantaged children with life-limiting illnesses living in the UK. They convert a massive percentage of the money donated to direct help for children (93%) and they get it done quickly.

You can donate using this link: http://www.smartgive.co.uk/funds/jungfrau/

Please give generously – we love you all
Mark, Rich and Andy

I had to recover a “legacy” document the other day which I only had a paper copy of. My typing skills aren’t bad until I have to look at a document rather than the screen or my fingers.

I often wish like an ex-girlfriend of mine I’d decided to learn to type properly – and before I get replies about how quick you are at bashing out lines of I’m talking about typing proper sentences that are on a sheet of paper next to you! I scored 61wpm with 1 mistake on this typing challenge which I don’t think’s too bad. And yes, I’m sure those of you with hummingbirds for fingers will be able to p*** on that. Anyway, she used to be able to type pages fault free nearly quicker than I could read them! I’d remembered that Clarky had looked in OCR technologies for a previous engagement and he recommended AABBYY Fine Reader as a good one.

I scanned in a few pages of the document to try out and it was pretty good (with a lot fewer mistakes than my typing). There were some pages that were an abject failure, including a flow chart that I had little hope of recovering anyway (and can be inserted as a GIF anyway), but most of them were on pages with smaller fonts. This might have worked better with a better scanner than the primitive one I was using. The only other real issues I found were some slightly quirky behaviour on bullet lists (didn’t recognise them), tables (where is made the tables but sometimes didn’t have a border on the LH side), and also occasional replacement of Ws with Vs (e.g. “Vat does this mean?”) so that the document sometimes looked like a bad ’Allo ’Allo script.

The software is fairly cheap at 89 quid for the download version and I’d be pretty tempted to convert the trial edition to the full version if I thought I wouldn’t begin OCR-ing inappropriate and irrelevant documents just for the hell of it (probably the modern day equivalent of labelling everything with Dymo tape machines) or testing it out on everybody’s handwriting.

I recently made a change to one of our products to switch from using a standard dropdown to using one with optgroups. The dropdown shows a Clients > Projects > Task combination so it made it much more readible to make the clients groups.

In IE/Firefox etc. the standard display is groups in bold and options in standard weight, either using italics or not dependant on the browser but for some reason Chrome makes everything bold.

The solution, via electrictoolbox, which is also somewhat strange is to turn off bold for the option group which actually turns it off on the child elements and leaves the heading bold – go figure.

To leave other browsers displaying entries properly too, there’s a final bit of css targetting via Martin Ivanov and things are back to the way they should be.

Caching

The final part of the prototype aggregator was to test out the caching capabilities. Caching for its own sake can cause pitfalls but in an environment like GAEJ, where every disk read costs, caching expensive data makes sense – even more so as GAEJ provides a robust cache implementation for you.

I wanted to cache the results from our news aggregator so the daily search results could be held in memory after the first retrieval. In order to do this I just needed to alter my data service to check the cache before going to disk. GAEJ uses the Memcache Java API which supports the standard JCache JSR-107 interface.

The nice thing about the implementation is that there’s no configuration needed. Each deployment in the GAEJ has access to a
javax.cache.CacheManager and you create a new cache through the following:

cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());

Then it’s just a case of putting to and getting from the cache … and identifying the lifecycle events for your cache items! But you’d have done that already, right ?!

In conclusion

The app engine is a great way to deploy and run Java web applications. The Eclipse plugin makes development and deployment very easy . It has a rich set of services under the hood. Caching, email, and scheduling are a breeze. The image manipulation is a little more involved, but still
much much easier than using other libraries or (perish the thought) writing it yourself. The persistence layer feels a little creaky after using Hibernate but is good enough for simple models. There’s a good management console and best yet, it’s free under certain limits! Google have just lowered these but we didn’t run in to any issues. I’d recommend trying it out … we certainly plan to develop products that can be deployed to the app engine. I’ll post to let you know how we get on with that.

It struck me the other day both how mobile devices are likely to drive improvements in the usability and accessibility of sites, especially those that want to give their users a similar experience on mobile browser and web browser.

Hiding functionality of information behind hover states and using a bit of dodgy javascript as a way to trigger them has always been a no-no but finding a way to convince clients who feel it makes a page more exciting is often difficult. With the advent of touch-screen mobile devices we can now side-step the discussion completely – no hover state whatsoever – joy!

Also anyone who has tried using their mobile browser to look at sites which pack in links and functionality without a thought for space, then found their Homer Simpson fingers have no chance of finding their target without opening adverts or clicking the wrong buttons, I’m in your club. Hopefully over time sites will start to add the necessary space and think about which buttons sit next to which.

From an accessibility point of view, I can see some of the features of screen readers would be useful in mobile browser pages, particularly information heavy ones. Pulling up a list of links or a list of headings would really help getting an overview of a page which requires a lof of scrolling. Also quick hand movements of key-strokes to trigger shortcuts would make a big difference. I’ll be keeping my eye on advances in mobile browsers to see if anything like that makes an appearance.

The app engine makes scheduling and email really easy. I wanted to take our news feed search, run it daily and then create a mailer that would send a digest to consumers each evening.

GAEJ has a cron-type scheduler built in, and creating scheduled jobs is as easy as adding a file called cron.xml to the classpath. Here is an
example file. You’ll see the syntax is much friendlier than the normal cron entries.

<cronentries>
	<cron>          
		<url>runSearches</url>
		<description>Run any searches required.</description>
		<schedule>every 12 hours</schedule>
	</cron>
	<cron>
		<url>/sendDigest?id=2001</url>
		<description>Send daily digest for West Ham news</description>
		<schedule>every 24 hours</schedule>
	</cron>
</cronentries>

Email support is taken care of through the standard javax.mail packages. There are plenty of good tutorials out there, like
this jGuru one on javasoft, so I won’t go in to the details.

A couple of things to note though. You don’t need to specify the mail host, but instead obtain it from the GAEJ properties:

Properties props = new Properties();

Session session = Session.getDefaultInstance(props, null);

…

Message msg = new MimeMessage(session);

Also, you need to make sure that you use the Google account you registered for GAEJ as the from address:
msg.setFrom(new InternetAddress("mygoogleaccountgmail.com"));@

And it’s as easy as that!

Last on the list of things to look at is using the Memcache API that GAEJ provides in order to cache content.

Next up was to enhance the prototype app by sending an HTTP request for an Atom output, parsing, and storing details from the results.

Sending the HTTP request was straightforward and used the java.net libraries. There’s plenty of resources out there about that so I’m not going to go in to it.

More “interesting” is the inevitable fun of dealing with XML in Java. In order to find the elements I was after in the Atom output I decided to use XPath as it’s a quick and fairly intuitive way of navigating a document and beats the getChildren() API calls of other Java libraries.

XPath support is within the javax.xml.XPath libraries. The specification requires abstract classes only, but Sun’s JRE provides an implementation in the form of com.sun.org.apache.xerces.internal.impl.xpath. However this library isn’t present in the Google JRE in the App Engine. This led to a runtime error
(when deploying to the App Engine but not in development) where it couldn’t instantiate an XPathFactory
(“XPathFactory#newInstance() failed to create an XPathFactory”).

In order to solve this, I needed to do the following:

• Include xalan in the project by copying the xalan jar to the /war/WEB-INF/lib directory
• Update the Eclipse build path to include this library
• Get a new XPathFactory by directly instantiating the Apache Xalan class:
  XPathFactory factory = new org.apache.xpath.jaxp.XPathFactoryImpl();
• Tell the App Engine to load the Xalan implementation by adding the following to the
  war/WEB-INF/appengine-web.xml file:
  

It’s slightly frustrating, but anyone who has tried to do anything using Java and XML will be familiar with the class loading hoops you have to jump through, so it’s not really an App Engine issue.

Anyway, following that I was able to retrieve and navigate the Atom output and store key pieces of information in the GAEJ datastore.

Next on the list will be scheduling and email …

I’m currently working on a project which is planning to make use of mobile devices to track where users are for tax and legal reasons. To help the explain the concepts to our client, I’m going to be create a set of wireframes of key screens in the user journey.

I have found a great set of stencils for iPhones from yahoo but after not finding anything for Blackberry, I created the my own Blackberry shapes for Visio

Just copy them into your My Documents > My Shapes Directory

Send back any comments/recommendations

Iteration 1 of the prototype was just to put together some simple objects and persist them in GAEJ’s datastore. There’s no relational database exposed via the app engine, instead persistent storage is implemented via
Datanucleus and accessed using JDO or JPA.

After fiddling around with JPA, I chose JDO as it seemed easier for the purposes. Don’t go thinking that you can use JPA and then switch your Hibernate code over, Hibernate isn’t really supported.

I created a simple class Search which owned a list of SearchResults and used annotations to mark up the attributes I wanted to persist. Then it’s just a case of using a PersistenceManager to manage the object’s lifecycle. I encapsulated all of these calls in to a SearchDataService to make defining transaction boundaries a bit easier. Having said that I did find issues in persisting relationships within a transaction which I wasn’t sure whether was down to dodgy configuration on my part or support for those features in the JDO implementation GAE4J. There’s a good tutorial on the Datanucleus JDO implementation here.

All went well until I wanted to change my List of SearchResults in to a Set (in order to make sure I didn’t store the same result twice).
At present, Datanucleus doesn’t seem to support Sets. When I implemented SearchResult as a Set I got a “Could not extract parent key from query expression” error. This is documented further at posting on the Google Group for GAE4J

It’s a bit of a limitation, but not the end of the world. It just means you have to do a bit of work before saving the results. Other than that it is pretty easy to get up and running persisting objects and deploying them to the app engine in the wild. The Admin Console that comes with
GAE4J is basic but allows you to browse and manipulate the data you’ve uploaded.

Next on the to-do list is using the app engine to connect to an HTTP API and parsing XML results.

I’m looking forward to testing out Google’s recent release of their App Engine for Java. Whilst it may not have the a full JRE there (check the
whitelist to see the supported classes) there is plenty there to build a full-featured application (persistence, email, scheduling, etc.) We’re looking to make a prototype version of one of our marketing search applications which should test out a lot of the features there.

The SDK comes with a nice Eclipse plugin (nice, if you use Eclipse as we do) which looks as if it makes development and deployment very easy. I’ll post more as I get through the prototype development and the rosy hue begins to fade from my spectacles :-)

I’ve always found sIFR frustrating to use (sorry Mike), typically having to fiddle with the end results to remove extra padding here or there which I could never find in my CSS. So on a recent project, when a colleague mentioned Cufon I thought I would give it a try.

The goals of the Cufon project sounded like exactly what I wanted – no plugins, compatibility with all major browsers, fast and close to zero configuration and after trying it, I would conclude that it’s achieved them.

The steps are really simple:

• Take your font file and run it through their converter to create a javascript equivalent
• Include the cufon.js file and the font file on your page
• Call Cufon.replace(’_selector_’)

and you’re done.

There are a couple of extras I had to look up. Hover states aren’t supported by default as they have performance implications, so use { hover: true } as the options in the second parameter to the replace call. Also you need one of the standard javascript libraries if you want to use more than just element selectors. I added jQuery – hey presto.

The one drawback is that text isn’t selectable or at least not in the same way standard text is or sIFR text is but if it’s used for headings or similar, it’s ideal!

We’ve been benchmarking and tuning the MySQL database for a decision support data mart product we’re currently developing and have had some interesting results.

The initial configuration that ships with the MySQL server isn’t necessarily suitable for every requirement(!) and so we’ve been experimenting with our lab server to try to find the optimal set up. (We’re using INNODB tables only so if you’re using MyISAM tables or a mix then there are different configurations that should be tested.)

We’re expecting data volumes to be fairly large for the data mart (i.e. millions of rows) and want to ensure that the it performs well. Our lab server is a lower specification that we’d recommend for the product but we have got some good results by tuning the buffer pool size and the query cache.

MySQL uses the buffer pool to cache both index and row data. In a denormalised data mart this is likely to give the biggest improvement in query times (as there usually is a lot of data in both!), and our lab database was no exception.

We managed to reduce the times of our benchmark queries by an order of magnitude when we set the size of the pool to 25% of avaiable memory. This is conservative given the usual recommendations to set to 80% but our lab machine also runs a web server. The buffer pool is configured through the innodb_buffer_pool_size parameter in the my.ini / my.cnf file.

We also saw a large improvement (several orders of magnitude) by enabling the query cache. The query cache stores queries and their results. It’s important to realise that it will only use a query from the cache if it’s exactly the same. Given that we are using named queries and will be querying data for the same month a lot of the time, it made sense to enable this. The query cache is enabled by setting the query_cache_size to a non-zero value.

We found it made the first run of the query slightly longer (as it puts the query in to the cache) but then subsequent runs very fast (e.g. 1800ms to 78ms). The trick is/will be to size the query cache to minimise the number of queries that are evicted (find this from the Qcache_lowmem_prunes variable).

We also use an application cache on the web server for our reference data so it won’t make sense to cache these queries as they will be run only once each server restart. Exclude queries from the cache by adding the SQL_NO_CACHE clause to the SELECT statement (e.g. SELECT SQL_NO_CACHE name, FROM person) but you will need to make sure that the query_cache_type system variable is set correctly.

As mentioned in the previous post the High Performance MySQL book has been very useful, as has the detailed documentation
on the MySQL site.

Last week, Ed Bateman and I went to a Make IT Scale seminar hosted by Sun and acardia in London. Most of the speakers were pretty good, but in particular Mike Griffiths from Proven Scaling gave an excellent whistle-stop tour around scaling MySQL.

At the moment, it’s a subject close to my heart as we’re designing a data mart that will back a Sales Decision Support System and we want our system to be future proof. As well as the tips he gave along the way, the main point that stuck with me is the amount of planning required to do the job properly. I’m already close to the time I originally allocated to the exercise and I reckon I’m around halfway through – and that’s after having done similar exercises before.

The one thing I didn’t remember him mentioning the need to resist the temptation to fix things using new features you’ve read about – i.e. before benchmarking and even knowing whether there is even a problem! It’s a real temptation, and psychologically very difficult to avoid. You read the new features, the use case in the documentation sounds just like yours and so it’s already a solution to a problem you might not have.

After the words of warning, it’s perhaps silly to recommend a book crammed with just those type of examples, but High Performance MySQL is by far the best book on MySQL that I’ve read. It takes you through the architecture of the product in fantastic detail and I reckon is a compulsory buy for people designing applications against the database. I’m reading it and enjoying it, and at the same time trying hard not to fiddle with the design and configuration until I know there’s a problem. I’ll let you know how I get on!

I’m a big fan of Miami Ink and although I’d never been sold on getting a tattoo myself I find the programme fascinating and some of the designs and art that the guys create are really amazing. Recently some of the team, Yoji, James, Darren and Chris, created Ruthless & Toothless which sells tattoo-art inspired clothing for kids, after not being able to find cool stuff for their own children. The designs are great and even better there are some other cool designs for laptops and phones. The iPhone covers are already giving me an itchy credit-card finger.

I wanted to take some time to identify what I feel are important steps that everyone should go through when creating a user interface. I firmly believe that usability is all about common sense and that at the heart of a good interface is a set of simple rules that are easy for the user to learn and remember so they can move effortlessly through your application.

I am going to use this post to look at the early stages of the design process. My assumption is that you will have been through some steps to identify the user and business requirements and the tasks/processes the user will go through to achieve them, be that through interviews, contextual inquiry or testing an existing system. If you haven’t, don’t let that stop you, it never usually stops anyone else.

The end point of this part of the process will be a set of wireframes and a screenflow diagram. Wireframes form an ideal end-point as they can happily feed into graphic design or be expanded on a little to form a testable paper prototype.

Before we go any further, I just want to capture what these two artefacts are to me as I know there is varied opinion on what they should contain and look like, so:

• Screenflow diagram – for me this means identifying single screens or pages in an application, showing how they are connected to each other (how the user can navigate or action and move between them) and identifying the goal or goals of each one. I see a screenflow diagram as being different from a site map as a site map is just a representation of the page/information structure. However, both could be used to document application templates.

• Wireframes – these I would define as a representation of pages of an application. They should show layout, component structure and functionality (either by using a series of pages to show functional states or using notes as descriptions). They do not show brand or design information.

So where to start .. as a first step I would mind map out the processes, user requirements and business requirements that you have received or gathered yourself. This should start to show you where there is commonality. I would consider each “task” as a page to begin with and then rationalise and consolidate them as a second step. When consolidating think whether you are adding confusion to any process or simplifying, whether you are creating a natural hub or stop off point. Also consider breaking out pages for specialist functions.

Once you have your pages, transfer them over into blocks on a screenflow diagram and start to link the flows. You will probably need a few iterations to get this right. Finish by describing the user goal on each page, this will help you understand if there is any room for confusion i.e. goals are duplicated, in conflict or woolly.

Once you have finished with all the standard pages, consider the homepage. It should have become clear what the key messages you want to communicate to the user are, what the start points for them are, these can form the basis for your homepage.

You are now in a position to create wireframes. This is where you should be thinking patterns and principles. When I say patterns I mean coarse grained ones – form filling, registration, inboxes/results pages rather than auto-complete and other finer grained ones. Coarse-grained patterns tend to get you greater standardisation and act as a helpful checklist. I will cover some in a future blog post.

As you add elements to your pages start to document a set of principles/rules and consider whether the element fits in with an existing principle or rule of your application or whether you’re creating a new one. I usually document these in a presentation so that I can deliver them with the wireframes. You’ll be able to read over them later and make sure none are in conflict and also review your wireframes against them.

At True North, we have a set of web application shapes to help us create wireframes in Visio. It saves us a lot of time, especially when you are using standard patterns. You can download them and use them yourself. Just save the shapes file into you’re My Documents > My Shapes directory then when you create a new document select File – Shapes – My Shapes – True North Wireframes.

After switching the layout to landscape, I always create a standard background that I use on every page. I include the browser container, the notes container and the document information container and then make that tab a background tab.

I use the natural visio grid to help me position elements. It makes my eyes bleed to see things not aligning to the grid lines and I believe that if you do it too often your page starts to look messy and confused so avoid it if you can.

So at the end of this process you should now be in possession of a brand-spanking new set of wireframes, principles and a screenflow diagram to hold them all together. Now you’re ready to go out and test them with your users. Good luck!

I hesitate to coin another “social” tag – I have a lot of time for Dennis Howlett and his take on the social meme stampede seems pretty accurate. However, after spending many of the recent years implementing workflow and BPM solutions, and hearing the recent exhortations from high to “do more with less”, it’s clear that traditional workflow and BPM implementations often find it hard to keep pace with process change, let alone reflect the real business processes.

Worse still, some implementations end up changing the business process to fit in with the technological constraints in some kind of twisted “build it and they will have to come” philosophy. I think there’s a real opportunity to use some of the tools and techniques of social media to create a workflow solution that better mirrors business processes and is adapts more easily to them changing.

A workflow solution usually has (at least) the following components: process definition; state management; role resolution; notification; metrics; tracking. The trouble with conventional workflow solutions are that they are a design-time solution – roles have to be assigned to users, process steps have to be designed and fixed, states have to be defined. Unfortunately, with budgets and departments shrinking, and supervisors often having to do the work as well as manage the processes and roles never stay that fixed. Also, it’s often only after you start trying to use a process that you see where improvements can be made.

Social media tools and practices are nothing if not fluid and “run-time” and they can be used to replace the parts of a workflow solution for ad-hoc and dynamic business processes.

Why not search users’ profiles to assign tasks rather than rely on a fixed role? (“I need a French speaker to check this company report”; “I need a director grade or above to approve this purchase”; “I need someone who has reviewed a vendor proposal before to look at this document”).

Why not use collaboration tools like group chat to track progress and team spaces to pick up and assign new tasks? People could also put themselves forward to act as participants with listed “surgery hours” to balance things alongside their day jobs.

Why not use tagging (social bookmarking) to identify common processes, best practices (or anti-patterns) in the process interactions or to group a set of smaller processes together in to a case?

Most of all, why not let the process initiator or participants decide the next steps in the flow and when it is complete? (“This document’s spelling needs rechecking”; “This person’s holiday is approved”; “This post is ready to be published”). This way we can get specific and directed tasks rather than “Reapprove”, “Complete”, “Reject”.

I’m not suggesting that social workflow works for all processes. There are a lot of very well-defined processes out there with strict roles, especially in highly regulated environments, and the need to execute and audit a set of steps and for this workflow and BPM solutions are ideal. Neither am I ignoring the fact that for some organisations this will be anathema at worst or require departmental change at best. But there are some processes and organisations where this is more appropriate and cheaper than implementing a workflow solution. In my experience this is a sizable majority of the processes implemented using workflow tools.

So the next time a workflow seems an ideal implementation of a business process, it’s worth asking a few deeper questions:

• Who really does the work? Does it fall naturally in to roles or is it more of a collaborative exercise? (A clue to this is whether the process has lots of different submit-reject-resubmit loops occurring in multiple places)
• How fixed are the steps? Are there really sequential steps or do people just work on it until it’s finished?
• How likely is the business process to change (and how exposed is this process to external factors)?

Answering yes to these might give you pause for thought that the solution could be better implemented as a social workflow solution.

This is the first of a set of posts where we’ll talk about our social media architecture. We hope to be able to add to the debate with prototypes or demo versions of some of the more bespoke components in the future.

Yahoo have recently added a number of social patterns to their existing Pattern catalogue. The patterns focus on Ratings and Reviews and Reputation. The reputational patterns are particularly interesting as a lot of them give rationale, links to related articles or on going discussions on the web. I’d would recommend that anyone with interest in usability and social media take a look.

I was lucky enough to visit the new Telenor offices recently as part of a business trip. They are built on the site of the old national airport at Fornebu, overlooking Oslofjord, the body of water that wraps itself around Oslo. The building is shaped like an eye, the old telenor logo, with two separate buildings forming the upper and lower parts of the eye, and the smaller separate building accessible via an overhead gangway is the centre.

When the sun is out and the sky is clear, the view is breathtaking and creates a really inspiring setting to work in and that was the goal of the building, to build an office of the future and a new way of working. Further examples of that goal come as you turn inward to face an expansive plaza, dotted with 92 space-odyssey style illuminated pillars, an art installation by French designer Daniel Buren. But perhaps the strangest piece of the building are the aphorisms that run along the top of the east building. They are the creation of Jenny Holzer and are quite dark when contrasted to the bright buildings, workers and setting. “I would kill her rather than watch a dirty ending but the killing would spoil my pity” was one uplifting example of what rolled by while we were taking a mid-morning break.