I’m looking forward to testing out Google’s recent release of their App Engine for Java. Whilst it may not have the a full JRE there (check the
whitelist to see the supported classes) there is plenty there to build a full-featured application (persistence, email, scheduling, etc.) We’re looking to make a prototype version of one of our marketing search applications which should test out a lot of the features there.

The SDK comes with a nice Eclipse plugin (nice, if you use Eclipse as we do) which looks as if it makes development and deployment very easy. I’ll post more as I get through the prototype development and the rosy hue begins to fade from my spectacles :-)

Iteration 1 of the prototype was just to put together some simple objects and persist them in GAEJ’s datastore. There’s no relational database exposed via the app engine, instead persistent storage is implemented via
Datanucleus and accessed using JDO or JPA.

After fiddling around with JPA, I chose JDO as it seemed easier for the purposes. Don’t go thinking that you can use JPA and then switch your Hibernate code over, Hibernate isn’t really supported.

I created a simple class Search which owned a list of SearchResults and used annotations to mark up the attributes I wanted to persist. Then it’s just a case of using a PersistenceManager to manage the object’s lifecycle. I encapsulated all of these calls in to a SearchDataService to make defining transaction boundaries a bit easier. Having said that I did find issues in persisting relationships within a transaction which I wasn’t sure whether was down to dodgy configuration on my part or support for those features in the JDO implementation GAE4J. There’s a good tutorial on the Datanucleus JDO implementation here.

All went well until I wanted to change my List of SearchResults in to a Set (in order to make sure I didn’t store the same result twice).
At present, Datanucleus doesn’t seem to support Sets. When I implemented SearchResult as a Set I got a “Could not extract parent key from query expression” error. This is documented further at posting on the Google Group for GAE4J

It’s a bit of a limitation, but not the end of the world. It just means you have to do a bit of work before saving the results. Other than that it is pretty easy to get up and running persisting objects and deploying them to the app engine in the wild. The Admin Console that comes with
GAE4J is basic but allows you to browse and manipulate the data you’ve uploaded.

Next on the to-do list is using the app engine to connect to an HTTP API and parsing XML results.

Next up was to enhance the prototype app by sending an HTTP request for an Atom output, parsing, and storing details from the results.

Sending the HTTP request was straightforward and used the java.net libraries. There’s plenty of resources out there about that so I’m not going to go in to it.

More “interesting” is the inevitable fun of dealing with XML in Java. In order to find the elements I was after in the Atom output I decided to use XPath as it’s a quick and fairly intuitive way of navigating a document and beats the getChildren() API calls of other Java libraries.

XPath support is within the javax.xml.XPath libraries. The specification requires abstract classes only, but Sun’s JRE provides an implementation in the form of com.sun.org.apache.xerces.internal.impl.xpath. However this library isn’t present in the Google JRE in the App Engine. This led to a runtime error
(when deploying to the App Engine but not in development) where it couldn’t instantiate an XPathFactory
(“XPathFactory#newInstance() failed to create an XPathFactory”).

In order to solve this, I needed to do the following:

• Include xalan in the project by copying the xalan jar to the /war/WEB-INF/lib directory
• Update the Eclipse build path to include this library
• Get a new XPathFactory by directly instantiating the Apache Xalan class:
  XPathFactory factory = new org.apache.xpath.jaxp.XPathFactoryImpl();
• Tell the App Engine to load the Xalan implementation by adding the following to the
  war/WEB-INF/appengine-web.xml file:
  

It’s slightly frustrating, but anyone who has tried to do anything using Java and XML will be familiar with the class loading hoops you have to jump through, so it’s not really an App Engine issue.

Anyway, following that I was able to retrieve and navigate the Atom output and store key pieces of information in the GAEJ datastore.

Next on the list will be scheduling and email …

The app engine makes scheduling and email really easy. I wanted to take our news feed search, run it daily and then create a mailer that would send a digest to consumers each evening.

GAEJ has a cron-type scheduler built in, and creating scheduled jobs is as easy as adding a file called cron.xml to the classpath. Here is an
example file. You’ll see the syntax is much friendlier than the normal cron entries.

<cronentries>
	<cron>          
		<url>runSearches</url>
		<description>Run any searches required.</description>
		<schedule>every 12 hours</schedule>
	</cron>
	<cron>
		<url>/sendDigest?id=2001</url>
		<description>Send daily digest for West Ham news</description>
		<schedule>every 24 hours</schedule>
	</cron>
</cronentries>

Email support is taken care of through the standard javax.mail packages. There are plenty of good tutorials out there, like
this jGuru one on javasoft, so I won’t go in to the details.

A couple of things to note though. You don’t need to specify the mail host, but instead obtain it from the GAEJ properties:

Properties props = new Properties();

Session session = Session.getDefaultInstance(props, null);

…

Message msg = new MimeMessage(session);

Also, you need to make sure that you use the Google account you registered for GAEJ as the from address:
msg.setFrom(new InternetAddress("mygoogleaccountgmail.com"));@

And it’s as easy as that!

Last on the list of things to look at is using the Memcache API that GAEJ provides in order to cache content.

Caching

The final part of the prototype aggregator was to test out the caching capabilities. Caching for its own sake can cause pitfalls but in an environment like GAEJ, where every disk read costs, caching expensive data makes sense – even more so as GAEJ provides a robust cache implementation for you.

I wanted to cache the results from our news aggregator so the daily search results could be held in memory after the first retrieval. In order to do this I just needed to alter my data service to check the cache before going to disk. GAEJ uses the Memcache Java API which supports the standard JCache JSR-107 interface.

The nice thing about the implementation is that there’s no configuration needed. Each deployment in the GAEJ has access to a
javax.cache.CacheManager and you create a new cache through the following:

cache = CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap());

Then it’s just a case of putting to and getting from the cache … and identifying the lifecycle events for your cache items! But you’d have done that already, right ?!

In conclusion

The app engine is a great way to deploy and run Java web applications. The Eclipse plugin makes development and deployment very easy . It has a rich set of services under the hood. Caching, email, and scheduling are a breeze. The image manipulation is a little more involved, but still
much much easier than using other libraries or (perish the thought) writing it yourself. The persistence layer feels a little creaky after using Hibernate but is good enough for simple models. There’s a good management console and best yet, it’s free under certain limits! Google have just lowered these but we didn’t run in to any issues. I’d recommend trying it out … we certainly plan to develop products that can be deployed to the app engine. I’ll post to let you know how we get on with that.

I was happy to see this blog posting which announces incoming email support for the App Engine. It definitely makes it possible to create more powerful applications with better legacy integration. We’ll be trialling out shortly and post here with our findings.

I was caught out the other day whilst writing a data service when I got a JDOException telling me I was accessing a field (in this case it was a serialized Collection) that hadn’t been detached.

This happens when you try to access an object, or collection within that object, which is still under the control of the PersistenceManager.

The PersistenceManager manages the persistence lifecycle of datastore objects, making sure they are synchronised with the underlying store. This is great in the data service layers, but most web applications will want to manipulate and display objects outside of this control before putting them back in to persistence-land. This is what detaching allows you to do.

The JDO spec is well worth a read as it describes the lifecycle in detail (a lot of detail :-)

There are several ways (including doing it manually) how you can handle the persisted object’s lifecycle and detaching in particular, but as we use transactions in our app the easiest was to use the JDO feature detachAllOnCommit which detaches an object and its collections. The code snippet below shows how.

Incidentally, although you can set detachAllOnCommit in JDO properties I prefer the explicit approach as it doesn’t hide potentially complex behaviour in configuration.

pm = jdoPersistenceHelper.getPersistenceManager();

pm.setDetachAllOnCommit(true);

tx = pm.currentTransaction();				

tx.begin();					

p = pm.getObjectById(Participant.class,participantId);				

tx.commit();					

pm.close();

...
// manipulate the paricipant here, for example
MailManager.invite(p.getEmail());

One final word about using Transactions in the Google App Engine. You can only enlist objects in a transaction that belong to the same EntityGroup. This means an object and its children (even objects of the same type are in different EntityGroups). We designed our application so that this would be easy for us, but it required careful thought and an app engine specific design. There’s more about transactions in the Google App Engine here which also addresses other ways to detach.

It took me quite a bit of searching and reading to find why although Google App Engine was using JSP 2.1 that my expression language expressions weren’t being evaluated.

You need to add

<%@ page isELIgnored="false" %>

into your JSP and then you’re away

It’s quite common to need to set an attribute value at runtime for a web application. Titles are a very common e.g. using a person’s name as the title for their profile page.

We’re currently using tiles for layout management in a GAE4J application so as it took me a little while to decipher the documentation, I thought I’d just write it up quickly here.

The steps are pretty simple:

• Create an implementation of ViewPreparer
• Link your ViewPreparer to your tiles definition.

A ViewPreparer is called before a definition is rendered.

Creating a view preparer

You only need to implement the execute method. The execute method gives you access to the TilesRequestContext which you can use to access request or session variables. They are where I would expect you to be pulling your dynamic attribute value from. An example is shown below

public void execute(TilesRequestContext tilesContext, 
    	AttributeContext attributeContext) throws PreparerException {
    	// get the session attributes
    	Map<String, Object> sessionScope = tilesContext.getSessionScope();
    	
    	Survey survey = (Survey) sessionScope.get(BaseServlet.SURVEY_ATTRIBUTE);
    	if (survey != null) {
    		String title = survey.getName();
    		
    		// if a name has been set, show it as the title
    		if (title != null && !title.equals("")) {
    			attributeContext.putAttribute(
    				"title",
    		        new Attribute(title));
    		}
    	}
    }

Linking the ViewPreparer to your tiles definition

Set the preparer attribute on your tiles definition. See the example below.

<definition name="survey" extends="admin" preparer="com.ontruenorth.socrates.view.preparers.SurveyTitlePreparer">
	...
</definition>

Enjoy!

Google announced a new version of the App Engine SDK today with a couple of interesting features.

Given the strength of feeling in evidence in the forums and blogs I expect it’ll be the raising of the 1000 limit on datastore queries that will get the most comment, but I was much more interested in the unit testing framework.

Unit testing support in the GAEJ has been non-trivial, which I suspect pretty much leads to non-existent testing in most apps. If the support makes this easier – especially in testing datastore access and caching – then it’ll be a hit with me.

There already was a way of simulating the app engine through a test classes supplied by Google but I’m hoping this will make things easier.

We’re going to upgrade the SDK and test out the new test framework and will post our results here.

We’ve just finished a presentation on tips and guidelines we’ve learned whilst working with the GAEJ.

It covers issues problems detaching objects, embedding associations, when to denormalise, and gives some advice on data modelling and when GAEJ may or may not be a good choice.

Take a look and we’d be really interested to hear your tips and experiences too.

The last post a good month and a half ago spoke about porting our revenue management product Jair to JRuby and deploying on Amazon’s Elastic Beanstalk.

Well, we’ve ported the app and apart from a few hiccups with gem versions not being supported in JRuby – we had to upgrade from the CSV to FasterCSV gem which I guess is no bad thing! – all went smoothly.

I really like JRuby as there’s something satisfying about writing in Ruby but deploying on to Tomcat. Even more satisfying is when the application actually runs up on Tomcat!

I’m not yet fully convinced as on shutdown Tomcat issues dire warnings about memory leaks being “likely to happen” but we’re going to do some application profiling soon so we’ll see.

Rather than use Elastic Beanstalk (only available in Amazon’s US regions at the moment) we just deployed on to a customised Bitnami Tomcat stack on AWS EC2. Again, this seemed a lot easier than the last time I tried although perhaps that’s down to having the right tools installed beforehand(!).

I’ve got a bunch of tips / things that we learned from the porting exercise and also AWS deployment so I’ll post those up in the next few days.

For us, AWS rather than GAEJ, is the way forward. Although there are different services on GAEJ (can only think of caching and IM offhand) so it could be useful for some apps, but how long before AWS offer the same? As well as that with AWS there is the freedom to install things like memcached etc.

I do feel a bit guilty though every day at 4p.m. when the True North chatbot (one of our apps on the Google App Engine) pings me on IM … maybe it’ll be its turn for porting next.