The Impala Blog

Dealing with library version clashes in Impala

2011-11-08T16:53:00.008Z

One of the problems that occasionally crops up in Java application development is library class version clashes. For some reason, you have to use a particular version of some library, but that library, or some of dependencies, clashes with other libraries that you are using for different parts of the application. I encountered an example of this occurring on our project, where we were forced to use the old Axis 1.4 library to connect with an integration partner. (For those who are interested, the reason is that the our partner is using the rpc/encoded SOAP API configuration, which is generally regarded as obsolete, and is not supported by any of the modern SOAP libraries, from CXF to Metro, as well as the Axis successor, Axis 2. See here and here for more on this problem.)

The ability to deal with these kinds of problems is often cited as one of the main reasons for using OSGi. In my development of Impala, I have steered clear from tackling these kinds of problems, on the basis that while they are theoretically quite valid to consider, in practice they occur quite rarely, and there generally is a workaround. Unfortunately, in my recent project, the workarounds seemed quite unpalatable. One would have been to resort a plain text based client for this SOAP API. For a small API, this would have been satisfactory, but for a large API it would have involved quite a bit of extra work. Ultimately, it seemed like a sensible point to add to Impala some capability for dealing with library version clashes.

The new mechanism simply involves the following:

add any module-specific library in the module's lib directory. For our project, this involved adding axis-1.4.jar and a couple of its dependencies
add the setting supports.module.libraries=true to impala.properties

This feature is currently in the code base and will be available in the next release, 1.0.2, which should be available soon.

How does it work?

In the normal Impala application, a class is loaded first by using the web application or system class loader (typically to load third party libraries), then using a topologically sorted list of the dependent module class loaders (typically to find application classes).

With module-specific libraries, the scheme is a bit more complex. For any module which has module-specific libraries, the module-specific library locations are searched first. Suppose we are loading module A, and we need to load the class Foo is in both the shared library area (WEB-INF/lib) and in a module-specific location for module A. In this case, the class Foo will be found in and loaded from the module-specific location.

Caveats

The main caveat is that the class Foo should never be 'published' from module A, for example as the return type in a method which might be called from another module, otherwise ClassCastExceptions or LinkageErrors can result. Instead, the interfaces to module A should ensure that appropriate abstractions are in place to ensure that this does not occur. In our Axis example, we need to ensure that instances of classes which hold references to Axis libraries are not passed around between modules.

More details on how the class versioning feature works, and how it can be configured, will be added to the documentation wiki.

The solution is by no means the solution to any kind of class versioning requirement which may crop up. Multiple versions for particular classes is still seen as the exception rather than the rule, and the scope as well as the complexity of the solution reflects this. That being said, there is at least a workable practical solution when you are stuck in a corner on this problem.

Impala 1.0.1 released

2011-08-10T21:53:00.006+01:00

I am pleased to announce the release of Impala 1.0.1. This release is basically a bug fix release. For more details on the bugs fixed, see this list.

While the pace of Impala's development has slowed in recent months, it is still in active development. I am still using Impala every day in my day job. If provides the foundation for the next generation e-commerce fulfillment software we are developing at Realtime Despatch. The reason I have been spending less time developing Impala is not because I have less use for it. On the contrary, it has really proved it's worth in the last year, making what could otherwise be a monster project quite comfortably manageable. The modularity and productivity features provided by Impala are not only fundamental for rapid development of our software, they also important in supporting our business model.

That being said, there are a few features that are backing up which I am hoping to develop in the next few months, and hope to pick up the pace again.

Impala Code Jam in London in London on May 26th

2010-05-20T16:21:00.003+01:00

Next Wednesday I will be holding an Impala Code Jam at Skills Matter in London, organised through the Java Web User's Group (JavaWUG). For the people who come along, this will be a hands-on opportunity to spend some time working on a simple Impala application and to get a better understanding of the concepts involved, and for most, to get a first hand flavour of how Impala can help in writing better, more flexible Spring based applications.

More details on the Code Jam are available from the JavaWUG announcement.

Some further comments on the 1.0 release

2010-05-12T10:10:00.000+01:00

I'd like to make some comments about the timing of the 1.0 final release of Impala. As I've mentioned elsewhere, the aim of Impala is to extract every last ounce of productivity and flexibility out of Spring-based development, while at the same time remaining true to the principles that made Spring popular in the first place - simplicity and testability. I believe that with this release I can confidently claim that these aims are being achieved, and that Impala provides one of the most productive and flexible environments for building Spring-based applications.

Impala has been used in real world projects for almost three years. Over this period it has undergone extensive refactoring to ensure that the architecture is right - flexible, maintainable, and open to the kinds of extensions which are likely to be introduced in the coming months and years. After over 5500 subversion commits, I am pleased to say that I am comfortable that this goal has been achieved.

While Impala has allowed me to achieve unprecedented levels of productivity in projects in which I have used it, I have also been mindful that more widespread adoption requires adding support to commonly requested features, even ones not necessary in my own projects. Much of the effort put into the project in recent months supports these objectives. Impala now has decent support for Maven, and also allows for fully modularised web applications, including their constituent classes, JSPs and resources.

This work has gone a long way to extending the capabilities of Impala and offering a greater choice in the technology combinations that will work out of the box with Impala. This work is continuing. Nevertheless, I am now comfortable that the feature set in Impala is suitable for a 1.0 release.

I have really enjoyed developing Impala and look back on the work with a lot of pride. It has been a fun project, full of technical challenges. However, there is still plenty of work to do, and plenty of exciting areas to explore. To this end I would very much welcome more direct involvement from the community in helping the project to achieve it's potential. This involvement can take any form you might imagine, from testing the framework in different environments, to building a web site for the project, to adding tool support, through to working on new samples and even new feature set.

Impala 1.0 final released

2010-05-12T09:38:00.007+01:00

I am pleased to announce the 1.0 final release of Impala.

Impala is a dynamic, modular productivity framework built around Spring. The aim of Impala is to extract every last ounce of productivity and flexibility out of your Spring development environment, while at the same time remaining true to the principles that made Spring popular in the first place - simplicity and testability.

Impala offers instantaneous redeployment of parts of your application, dramatically improving build/deploy/test cycle times. Impala supports on the fly 'drop-in' of new application functionality - including web modules - without requiring an application restart and without requiring any changes to existing application code or configuration.

With Impala you can achieve the productivity benefits of a modular approach to application development. When using Impala, you no longer need to work with application contexts which contain hundreds of bean definitions. Instead, you break your application down into smaller, manageable chunks. You no longer need to put up with the ever-increasing complexity and tangled interdependencies which come with a monolithic approach to application development. Instead, you develop your application in a loosely coupled way, with a clean distinction between application's interfaces and implementation components.

With Impala it is much easier to mix and match features for particular environments, which is particular useful for applications which may need to be deployed with different features and configurations on a per-environment basis. Writing integration tests with Impala is a doddle, because setting these up is simply a matter of selecting the modules you want to include in your tests.

Impala works with all the technologies you would expect in the Spring world - Hibernate, Quartz, JMX, etc. - without requiring any special plugins to be written for those technologies. It also supports - in most cases without modification - web applications using a variety of web frameworks, from Struts to JSF to Tapestry and others. A lot of work has been put into making Impala elegantly support modular web applications, allowing for modular vertical web application 'slices', rather than just a modular back end.

Impala works straight out of the box using a plain Eclipse installation (no fancy plugins required). It has a built-in build system which you can optionally use, based on Ant, and also integrates nicely with Maven.

With Impala, you can take your Spring-based application development to a new level of ease and sophistication, with remarkably few changes to the way you write applications.

See the full list of issues covered in this release.

With Impala 1.0 released artifacts are also available in the Maven central repository. See http://repo1.maven.org/maven2/org/impalaframework/.

If you like Impala and would like to support the project, please take a look at this page: http://code.google.com/p/impala/wiki/GetInvolved.

Impala 1.0 RC4 released

2010-04-25T19:56:00.002+01:00

I am pleased to announce the release of Impala 1.0 RC4.

Impala 1.0 RC4 was originally supposed to be the 1.0 final release. However, due to a few more than expected changes in this release, it was decided to have one last release candidate prior to 1.0 final.
The main changes in this release are:

simplifications and enhancements in the mechanism for mapping web requests to modules, allowing for more intuitive arrangements of JSPs and resources within modules.
modification of the starter application to use the multi-web module mechanism out of the box, as this is the recommended approach for building Impala web applications.
the addition of Javadocs to the build artifacts (this is now required for artifacts deployed to the Maven central repository via Sonatype's infrastructure).

For more information on this release see: http://code.google.com/p/impala /wiki/Release1_0RC4Announcement.

Phil Zoio

Impala Home: http://impala.googlecode.com

Seven ways in which Impala makes Spring development more productive

2010-04-05T21:06:00.000+01:00

Impala is a modular, dynamic productivity framework. The principal aim of Impala is to help developers extract every ounce of productivity out of the Spring development environment, while remaining true to the original Spring premise of simplicity and testability. The result is a much more productive Spring application development than you might think possible.

Here's how Impala lets you be more productive when working with Spring.

1. Virtually instantaneous build/deploy/test cycles

For typical Java applications, redeploying a change requires redeploying the whole application. Not with Impala. Only the affected parts of the application need to be deployed. This leads to virtually instantaneous build/deploy/test cycles, as the granularity of redeployment is typically very small relative to that of the application as a whole. You don't need to sit around for ages waiting for your whole application to load.

2. Automatic modification detection and redeployment

When you make a change to an Impala web application running in Eclipse, Impala can automatically pick up which parts of the application have changed, and redeploy those parts of your application without any user intervention. This removes the need for a manual step to perform this task. Impala gives you control over what resources are monitored for changes, how frequently resources are checked for modification, etc.

3. No explicit build step required

When writing typical Java web applications using Eclipse, in many projects there is an extra build step required to package your applications in order to deploy them. Not with Impala. Eclipse automatically compiles your classes for you incrementally in the background. The project structure conventions for Impala mean that this sufficient, and that no extra build steps are required during development.

Of course, you can do a regular WAR build when required for packaging for a production environment.

4. You can write your integration tests interactively

Writing Spring integration tests can be slow, particularly for monolithic single module applications. Impala allows you to write integration tests interactively. You can write and run integration tests using an interactive console within Eclipse. If making changes to your application between test runs, you simply redeploy the changed modules. If you're simply changing your test, you don't need to do anything special as changes to your test class will automatically be picked up on the fly. All of this leads to a much more rapid build/deploy/test cycle when writing integration tests than would be possible otherwise.

5. Suites including integration tests run super-efficiently

When running a suite of integration tests, modules needed by successive tests are loaded incrementally and remain available for subsequent tests run as part of the same suite. There is no need for continually starting and stop Spring application contexts, which can really slow down the execution of integration tests.

All of this makes test suite execution really fast, making Impala very friendly to use in environments where developers are expected (as they should be) to run the full project test suite before each commit to the version control system.

6. Setup is very simple

For a developer getting started on an existing project using Impala, it is really as simple as "check out and go" - simply check out the project files and starting writing and deploying the application within the IDE.

If you're starting a new project, this too is easy, as Impala provides a script which allows you to set up a simple, working application by executing a couple of commands.

7. Easy to use build and dependency management support if you need it

Impala works with Maven, but if you're not a fan of Maven, Impala also features a reusable build system which saves you having to write create your own build from scratch. Impala also gives you a simple way to pull third party library jars and source from Maven repositories, even if you aren't using Maven as your build system.

Impala 1.0 RC3 released

2010-03-12T14:37:00.004Z

I am pleased to announce the release of Impala 1.0 RC3. Impala 1.0 RC3 is the final release candidate prior to the official 1.0 release. If no significant issues are raised against this released, it is envisaged that there will be virtually no changes in the subsequent 1.0 final release.

This release contains a number of enhancements:

simplifications making it easier to migrate existing Spring-based projects to use Impala. It is now possible to run Impala as part of an existing virtually unchanged traditional Spring web application, which should significantly simplify the task of migrating an existing Spring application to use Impala.
support for JSPs deployed within individual web modules. Prior to 1.0 RC3, JSPs need to be contained within the web application folder. With 1.0 RC3 it is now possible to completely modularise the web elements of a JSP-based web application.
support load time weaving of aspects created using AspectJ.
various other minor features and bug fixes.

Also, Impala 1.0 RC3 is available in the Maven central repository for the first time.

For more information on this release see: http://code.google.com/p/impala/wiki/Release1_0RC3Announcement.

In the next couple of weeks, I plan to add more documentation on how to get Impala working nicely with Maven, and how to migrate an existing Spring-based project to use Impala.

Please don't hesitate to report any issues you may encounter, which you can do on the Impala issue tracker: http://code.google.com/p/impala/issues/list.

Phil Zoio

http://impala.googlecode.com

OSGi and Enterprise Java dynamic modularity after Spring dm Server

2010-02-03T07:12:00.001Z

The key message to be taken from SpringSource's decision to hand last month their Spring OSGi projects (Spring DM and dm Server) to the Eclipse foundation is the following: that OSGi is not ready for adoption by the mainstream of enterprise Java. Reality has finally dawned!

According the announcement from Adrian Colyer, CTO of SpringSource:

For a mainstream development team though, who just want to build an enterprise application as quickly as possible, and with as little hassle as possible, the costs currently associated with adopting enterprise OSGi can outweigh the short-term benefits. This situation needs to be addressed before enterprise OSGi can become the de-facto approach for mainstream enterprise application development.

While SpringSource will no doubt continue to support OSGi, the extent to which it will continue to provide resources to the donated projects is far from certain. What is more clear, though, is it will no longer be wielding it's own considerable influence within the industry to shepherd its user base en masse towards OSGi.

This is a good thing. Firstly, the obsession with OSGi has distracted the community away from practical efforts which could work more comfortably within the existing enterprise Java model. Hopefully that obsession will go away, or at least be limited to those already converted to the OSGi way. Secondly, from a purely selfish, egotistical point of view, I feel vindicated. For years, I have been saying that OSGi is too complex for the Java mainstream. For years, I have been beaten over the head by a small group of hardcore OSGi evangelists who have persistently used the argument that if SpringSource is betting on OSGi, then it must be a good thing for everyone else, and that any alternative approach does not warrant any consideration. The bet has failed, and this argument has now fallen rather flat. If SpringSource, with all the credibility it has generated through the Spring Framework, all its intellectual and marketing muscle, and all the effort and expense it has incurred, cannot make work OSGi for mainstream enterprise Java, then no one else can.

But all of this is also points to a real missed opportunity. The fact remains that Spring users still need a modularity solution. They need a way of being able to take advantage of the best of what dynamic modularity has to offer - dynamically composable software with genuine decoupling between parts of the application, and rapid build/deploy/test cycles during development - without having to take on all the complexity of the full-blown OSGi approach. This need still exists, and will not go away. As Adrian suggests, this need will not be served by OSGi without some fundamental simplifications, something I could not imagine happening any time soon.

In the meantime, I have invited SpringSource to engage more with Impala. While they have flirted with the idea in the last few months, perhaps even seriously, they have backed away from it for now. In view of recent history, it is not that surprising. Imagine the uproar that would ensue among the OSGi zealots if SpringSource were to go down that route. It doesn't bear thinking about.

Impala 1.0 RC2 released

2010-01-04T14:54:00.003Z

I am pleased to announce the release of Impala 1.0 RC2. With this release, support for Maven is introduced for the first time, along with various other enhancements, refactorings and bug fixes, including:

simplifications to the default project structure, making it more intuitive and Maven-friendly
a dynamic properties Spring namespace, making it easier to support properties which can be injected and dynamically updated without requiring module or application reloads.
further refinements to the mechanisms for supporting multi-module web applications.
initial support for Spring 3.0.

For more information on this release see: http://code.google.com/p/impala/wiki/Release1_0RC2Announcement.

Cheers,
Phil Zoio

Don't let OSGi put you off modularity

2009-11-26T08:51:00.001Z

If you are interested in the ideas of modularity, there is quite a good chance you have looked at OSGi. But there is also quite a good chance that you have been put off by the extra development overhead required to set up and maintain OSGi-based applications.

If this is the case for you, my message is this: don't let fact that you are put off by OSGi put you off the ideas of modularity. Modularity is a bigger, more important and valuable idea than any particular technology. The ability to modularise an application can massively improve your application's ability to absorb complexity as new features and components are added, which in turn means that you remain much more productive as your application grows. It reduces the rate of entropy in your system, extending its potential life span, which of course if great for business.

This kind of modularity is one of the fundamental benefits of Impala. Also, because Impala's modules are dynamically reloadable, and you also get dynamic redeployment of small parts of your application, allowing you to greatly accelerate application development, maintain high levels of productivity, and retain high levels of code and functionality reuse as you application grows.

Impala does not solve all of the problems solved by OSGi. Notably, it does not provide versioning of third party libraries. However, it solves most of the important ones relevant for day to day development. Fundamentally, it allows you to modularise your application, without having to worry about micro-managing the dependencies of third party libraries in your application. Don't get me wrong - there are times when this capability can be important, especially for very large projects with large budgets and teams. But certainly not for every project, and probably not even for the typical one.

One way to think of choice is to look at the graph below.

The basic idea behind modularity is that the growth in complexity of your application slows as your application grows in size, compared to applications without modules. This applies both for Impala and OSGi-based applications. The difference is that because the barriers to entry are lower for Impala-based applications, the benefits kick in sooner, and accumulate for longer over the duration of the project, greatly reducing the overall cost of complexity over the lifetime of a project.

So, don't shy away from modularity just because OSGi looks complex. The benefits of modularity are too valuable, and an alternative like Impala make these attainable with fewer headaches.

The granularity of change in dynamic Java web applications

2009-11-05T20:19:00.002Z

When writing Java web applications, you are continually making changes to your application, and to be productive you need to be able to deploy and test these changes quickly. The kinds of changes you make are of all sorts: from changes to resources to changes to markup templates to changes in the way your application is wired to changes in the code itself.

The point of this article is that not all changes are equal, both in their frequency and in the difficulty in applying them dynamically in a running web application. Let's go through some examples, and how the changes might be applied in different types of frameworks:

1. Static resources
The simplest kinds of changes for any dynamic web framework to apply are those of static resources - images, JavaScript files, page templates etc. That's because these kinds of artifacts are inherently free of dependents.

2. Application configuration
Configuration consists of things such as settings for data sources, mail servers, etc, as well as switches in your application itself. While it is in general possible to reflect these kinds of changes dynamically, there is a cost. For example, if you are using database connection pooling, pointing to a different data source dynamically is a non-trivial exercise. Also, if your application checks particular application settings at source each time the affected functionality is used, then the system is more dynamic, but also less performant. By contrast, if you only load particular settings at startup (for example using wired in property placeholders in a Spring application context), the application is more efficient but is more likely to require reloads to reflect changes.

3. Application wiring
Application wiring is a configuration of sorts, but relates more to how parts of the system are composed or wired together to form the whole application. In a Spring application, the application wiring is simply your Spring configuration.

In general, changes to application wiring requires reloads. There are special cases where this doesn't apply. For example, you could introduce new Spring beans without reloading the application context. Changes to existing beans and their collaborators are harder to make.

4. Scripts
Scripts are programs in your application which by definition can be altered without having to reload your entire application or even significant parts of it. However, a scripting infrastructure needs to be in place to allow changes to scripts to be introduced, recognised and reflected in the system.

5. Application code
Application code in a Java application are your Java classes. Actually, considering this group as a single category is an oversimplification, especially in a dynamic module system, where making changes to core interfaces and domain classes will impose much greater requirements in terms of reloading than peripheral or implementation classes with fewer dependants.

6. Third party libraries
The libraries in your application are the jar files containing all the third party dependencies.

The challenge for frameworks

The key productivity challenge with a dynamic application framework is to make it as easy as possible to make the kinds of changes you need to make, while at the same time keeping the framework as lightweight as possible. In the next section I take a look a number of technology stacks, what they do to make different types of reloading possible, what they get wrong, and what they get right.

A. Traditional Java web application
A traditional Java web application might consist, for example, of a Struts or JSF front end, Hibernate back end, all wired together using Spring.

The traditional Java web application has no problem reloading static resources without having to reload any other part of the application. Most web containers are able to reload an entire web application, including third party libraries, Java code, application wiring etc.

The problem is they are not much good at reloading any finer grained changes. Any changes you make to your application wiring or code will normally require a full application reload.

B. Scripted applications
Scripted applications are based on scripting languages such as Groovy (Grails) and JRuby (Rails). As well as explicitly providing support for reloading capabilities, these frameworks rely on the fact that all application functionality is in scripts rather than in compiled Java code, making fine grained reloading of parts of the application possible. The downside (if you think of it this way) is that you have to work with scripted code without any of the type safety checking of a statically typed language such as Java.

C. OSGi applications
OSGi applications offer a fairly comprehensive solution to the reloading problem. All artifacts within the application, from resources to application code to libraries are contained within modules which are treated in a more or less uniform manner by the OSGi container. This is a strength, but it is also a weakness. The strength is that it does allow third party libraries to be reloaded in a fine grained way. The weakness is that in your high level view of the application, OSGi doesn't really allow you to easily distinguish between the parts of your application which should be easy to reload - e.g. resources - and the parts which are harder to reload, but are changed much less frequently during the lifetime of the application (third party libraries).

What about Impala?

Impala tries to find the right balance in the strengths of the various approaches. Resource reloading works as with traditional Java applications - nothing special needs to take place. Impala includes mechanisms which make it easier to change configurations dynamically without requiring any module reloading. For changing static configuration and application wiring, Impala allows you to reload parts of your application at exactly the right granularity. If only a single module is affected, then only that module needs to be reloaded. If the change affects core interfaces, then the reload will automatically ripple through to the right dependent modules.

Impala even allows you to dynamically change the structure of modules within the application. Unlike OSGi, it doesn't support reloading of third party libraries. For this, an entire application reload is required. However, Impala's approach does your application modules in central focus, which is important as these are the normally parts of your application which change most frequently.

Slides for Impala talk at the Server Side Europe in Prague

2009-11-01T20:32:00.004Z

I'm pleased to say that my talk at The Server Side Java Symposium in Prague went well and was apparently well received. There definitely is a growing interest in the ideas of modularity and how the benefits of modularity can be achieved in practice.

I've posted a copy of my slides for the talk here in PDF format. The document contains the slides that I presented, as well as quite a few which were held in reserve but weren't actually presented during the talk.

How to make your Spring wirings more manageable

2009-10-14T20:15:00.002+01:00

When Spring first came along it was a breath of fresh air - a clever way to wire up applications which did not rely on the use of all sorts of singletons all over the place. (I still remember what it was like working that way, and shudder at the thought.) The idea was simple: let classes in the application just do their own job, but leave the business of figuring out how to get collaborators to the IoC container, with the help of some XML configuration. No longer did your application code did not have to deal the messy business of having to resolve their own dependencies.

OK, we've solved the problem with the code, but the job isn't completely done. Actually, the problem has shifted onto managing Spring wirings.

The problem with managing Spring wirings

Applications necessarily get big. You end up having to write a lot of XML. So while your code may stay nice and clean, you end up with some tricky questions about how to manage this part of your application. Of course, here you try to be as "modular" as you can, putting DAOs together, infrastructure related beans together, etc. You try to identify vertical slices for your application and put beans relating to particular vertical slices together.

The problem is that for all the bean definitions that exist in your application, some groups are inherently coupled, while others are inherently free of coupling. In a vanilla Spring application, there is no way to express these dependencies at a system level. So it is very easy for your application wiring to become an unnecessarily fragile collection of invisible dependencies, liable to break in unexpected ways when any rearrangement takes place.

Autowiring, namespaces and class path scanners don't necessarily help

Then of course there is the drudgery of editing XML configuration files by hand. Personally, I think that is less of a problem, but Spring has gone to great lengths to free developers from some of this pain over the years, through the introduction of autowiring, class path scanners, and XML namespaces. I happily embrace all of the above as they reduce the amount of code I need to write, but they don't address the fundamental problem. They don't enhance one's ability to express dependencies between parts of your application at a system level, and where possible, to reduce these dependencies.

So how does Impala help?

Remember how easy Spring seemed when we were working with just small applications? Impala allows you to keep your applications small, or at least keep them feeling small. This is done through modules. You can think of a module as a mini-application which is able to communicate with other mini-applications in the system through well defined mechanisms and through sharing common interfaces.

The Spring configuration for each module remains pretty small. If it starts getting too big, then its a good sign that some of it's functionality needs to be split off into another module. So within the module, you only need to deal with small configurations - bite size chunks.

You can configure beans within a module however you like - through plain Spring XML, custom namespaces or through annotations. If your module needs to use functionality from other parts of the system (as most will), then you can import services directly from the shared service registry (as long as the service has been exported using an interface or class visible to the current module). If necessary, you can allow your module to depend directly on another module, either as a dependent or as a direct child.

If you need to compose your application in different ways according to environment or customer, that's easy too. Simply change which modules your deploy, or you can even vary the configuration of within a module according to requirements.

You no longer need to wait ages for integration tests to load, because you can easily create integration tests which consist simply of the modules you need to use.

And you get the benefits of much more productive, responsive development environment because each of these modules can be reloaded on the fly, either individually, in groups, or as a whole - and this applies whether you are running integration tests or running your application on a web container.

Talk at the Server Side Europe in Prague

2009-10-06T08:58:00.002+01:00

I am doing a talk on Impala at The Server Side Europe's conference in Prague, which is taking place on October the 27th and 28th. Really looking forward to it, especially as I used to be a regular visitor of Prague in the early 90s when I was living in Germany.

Why web.xml makes it hard to write modular web applications

2009-10-03T14:52:00.000+01:00

In a typical Java enterprise application, the web.xml is used to define servlets and filters, which are among the main entry points into your application from the outside world. Since web.xml cannot be reloaded, added to or modified without reloading the entire application, it is not a very convenient place to host application configuration and definitions in a dynamic module applications.

Another related problem is the limitations of the request mapping capability of web containers as defined by the current servlet specification. Currently, these make it possible to map requests to servlets and filters using an exact match (e.g. /myservlet/myresource) or either a prefix and wildcard match (e.g /myservlet/*) or using a suffix wildcard match (e.g. *.do). It doesn't allow you to use a combination of prefix and suffix wildcard matches. This means that you cannot, for example, use the path (/myprefix/*) to match application URLs, and at the same time allow your application's CSS files to be accessible in a resource such as /myprefix/styles.css.

Multi-module web applications in Impala

One of biggest changes in the recent 1.0 RC1 release of Impala is the ability to write web applications which are less reliant on web.xml, allowing both dynamic registration of modules containing servlets and filters, and at the same time solving the path mapping limitation described in the previous paragraph.

In an Impala application, you cannot do away with the web.xml altogether. However, you can reduce the request handlers defined in web.xml to the following:


<filter>
<filter-name>web</filter-name>
<filter-class>org.impalaframework.web.spring.integration.ModuleProxyFilter</filter-class>
<init-param>
  <param-name>modulePrefix</param-name>
  <param-value>urlmapping-web</param-value>
</init-param>
<load-on-startup>2</load-on-startup>
</filter>

<filter-mapping>
<filter-name>web</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

In the example above, the ModuleProxyFilter captures requests and routes them into Impala modules. The mapping rules which determine which modules service which requests are contained within modules. Here's an example from the URL mapping sample:


<web:mapping>
<web:to-module prefix = "/webview" setServletPath="true"/>
<web:to-handler extension = "htm" servletName="urlmapping-webview" filterNames = "characterEncodingFilter,sysoutLoggingFilter"/>
<web:to-handler extension = "css" servletName="urlmapping-resources"/>
</web:mapping>

<web:servlet id = "urlmapping-webview"
servletClass = "org.impalaframework.web.spring.servlet.InternalModuleServlet"/>

<web:servlet id = "urlmapping-resources"
servletClass = "org.springframework.js.resource.ResourceServlet"
initParameters = "cacheTimeout=10"/>

<web:filter id = "characterEncodingFilter"
filterClass = "org.springframework.web.filter.CharacterEncodingFilter"
initParameters = "forceEncoding=true,encoding=utf8">
</web:filter>

<web:filter id = "sysoutLoggingFilter"
filterClass = "org.impalaframework.urlmapping.webview.SysoutLoggingFilter">
</web:filter>

This module defines a number of servlets and filters who's life cycles are tied to that of the module, rather than that of web.xml. They can be dynamically registered and removed, and don't require an application restart. The modules can contain all the classes and resources necessary to service requests, without relying on the presence, for example, or resources such as JavaScript files on the context path (e.g. in the WEB-INF directory).

What about Servlet 3.0?

The changes described above are very much in line with the changes in the forthcoming Servlet 3 specification which allow servlets and filters to be added via web.xml fragments, and via annotations. It will also allows you to add Servlet and Filter instances programmatically. I expect that Impala will be able to take advantage of this mechanism when it becomes available, perhaps by wrapping the Servlet or Filter to ensure that it is associated with the originating module's class loader, and not the web application class loader. This will have the advantage of allowing Impala to make use of the web container's invocation infrastructure while still supporting dynamic servlet or filter registration.

Impala 1.0 RC1 released

2009-09-24T14:17:00.006+01:00

I am pleased to announce the release of Impala 1.0 RC1.

With this release, Impala is now feature complete for the 1.0 final release, with only minor enhancements and bug fixes planned before this happens.

Impala 1.0 RC1 contains a number of big feature improvements from the previous release:

a more powerful and flexible mechanism for mapping web requests to and within modules, making it easier to build truly multi-module web applications.
a new Spring web namespace for registering servlets, filters and other web artifacts in Impala web modules.
enhancements to make the automatic module reloading mechanism more robust and suitable for applying in production environments.
various other minor bug fixes and enhancements, particularly in the areas of build, dynamic services and class loading.

For more information on this release see http://code.google.com/p/impala/wiki/Release1_0RC1Announcement.

Enjoy!

10 reasons for Spring users to try Impala

2009-09-06T19:13:00.045+01:00

If you are a user of the Spring framework and you haven't tried Impala, I hope to convince you in this entry that you are really missing out.

These are the reasons why I make this claim.

When Spring was introduced 2004-2005, it brought a big shift in the frontier of Java enterprise software development, offering solutions to many of the challenges faced by developers in a way which other technologies before, notably EJB, had conspicuously failed. Spring is still very much a valid technology in today's environment, but the frontiers have shifted.

One shift is in the increasing recognition of the shortcomings of the Java language itself, and the need for an eventual replacement as the premier JVM language. Another shift is the increasing awareness of the need for modularity in application development. My view is that without the backing of a truly modular framework, it is almost impossible to build a large enterprise application which does not end up becoming unwieldy, difficult to manage, and slow to build and deploy. This more than anything else leads to the perception that Java is unproductive to work with.

1. Impala gives your project a massive productivity boost
The reason for this stems from Impala's dynamic reloading capability. When you make changes to an Impala application, you only need to reload the affected modules and their dependents. Indeed, you can set Impala up so that these modules will be reloaded automatically. This allows code/deploy/test cycles to be reduced to an absolute minimum. In some of the next few points we give some further examples of how Impala improves your productivity.

2. With Impala you can make your applications truly modular
Modularity is about removing all unnecessary coupling between parts of your application, and is essential for building large applications which don't grow exponentially in complexity as they grow linearly in size.

With Impala and other dynamic modularity frameworks, modularity is achieved through a separation of interface and implementation which is just not possible in traditional Java applications. With Impala, modularity is enforced at the class loader level.

Impala makes it simple to deploy multiple flavours of the same application simply by choosing which modules to deploy. Impala gives you mechanisms for configuring individual modules, so that you can support different deployment options in a clean and simple way.

3. Impala works "out of the box"
I can't resist using this term because it features often in Rod Johnson's books. If you try any of the Impala examples, you will notice that all you typically need to do is to check out the code, then run it in your IDE. (If a database is involved, you may need a little extra setup there.) There is no need for any extra build steps, installation of third party environments, containers, etc. It's all self-contained.

4. With Impala, Spring integration testing is a doddle
Impala makes it really simple to write Spring integration tests. Gone is the need for convoluted test application context definitions such as


new ClasspathApplicationContext(new String[] {
"config-context.xml",
"dao-context.xml",
"config-context.xml",
"some-context-which-you-you-need-for-your-test.xml",
"some-context-which-you-you-dont-need-for-your-test.xml",
"another-context-which-you-you-dont-need-for-your-test.xml",
}

With Impala, all you need to do in a typical integration test is to specify which modules you want to include (dependent modules get included automatically), and use Impala's API to access beans either from the root or one of the module application contexts. For example:


public class InProjectEntryDAOTest extends BaseDataTest {

public static void main(String[] args) {
  InteractiveTestRunner.run(InProjectEntryDAOTest.class);
}

public void testDAO() {
  //get the entryDAO bean from the root module
  EntryDAO dao = Impala.getBean("entryDAO", EntryDAO.class);
  ... test methods
}

public RootModuleDefinition getModuleDefinition() {
  return new TestDefinitionSource("example-dao", "example-hibernate").getModuleDefinition();
}

}

Tests work equally well whether you are running them interactively (without the need to reload the entire application or even any part of it between successive test runs), or whether you are running it as part of a suite (in which case the application modules are incrementally loaded as required). In both cases, running integration tests is very efficient, allowing you to be much more productive while still practicing TDD.

5. With Impala you can write truly multi-module web applications
A big barrier to truly modular web applications is the reliance on web.xml, because changes to web.xml require a full application reload. The forthcoming release of Impala allows you to define servlets and filters within the modules themselves, with their life cycle tied to that of the containing module. It also allows you to map requests with arbitrary URL path prefixes to individual modules. Once within a module, requests can be mapped to filters and servlets based on file extension.

These capabilities allow you to create a web application tier which is truly modular: you can package all of the filters, servlets, controllers, templates, images, Java script files required to service URLs with a particular prefix into the module jar itself. Indeed, you can even reduce your web.xml filter and servlet declarations to the following:


<filter>
  <filter-name>web</filter-name>
  <filter-class>org.impalaframework.web.spring.integration.ModuleProxyFilter</filter-class>
  <init-param>
          <param-name>modulePrefix</param-name>
          <param-value>urlmapping-web</param-value>
  </init-param>
  <load-on-startup>2</load-on-startup>
</filter>

<filter-mapping>
  <filter-name>web</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>

See the web.xml from the URL mapping sample for the full example.

6. Impala does not reinvent the Spring programming model
Spring has been a huge success for a good reason - because it offers a much simpler programming model than was offered before. Spring offers a great solution for dependency injection as well as for making otherwise key technologies - transactions, AOP, JMX, remoting, etc - accessible or very simple to use.

Impala does not reinvent the Spring programming module - within modules you will recognise all the artifacts familiar to Spring applications: collaborators wired together via dependency injection, configured using XML configuration files, annotations, etc. The big difference is that Impala gives you a well defined way for expressing relationships between modules within the application, allowing for simpler implementations of - and clear boundaries between - the constituent parts of a bigger application.

7. With Impala you can forget about the build (most of the time)
Impala allows you to develop your applications in a vanilla Eclipse environment without having to invoke any build scripts or do any extra environent setup to run your application. This makes getting new developers started on projects extremely simple. Simply check out and go.

Of course, you will need to build for production environment. Impala includes an ANT-based build system which allows you to build either a WAR file or a Jetty-based standalone web application, which you simply unzip in the target environment and run.

8. Impala fundamentally simpler to use than OSGi-based alternatives
OSGi is a powerful technology, a very complete modularity solution. It is also quite a complex technology, requiring a non-trivial investment in time and energy to understand it, both conceptually and in its effect on the application environment.

From a practical point of view, applying OSGi to enterprise environments is far from trivial, and involves solving a number of challenging technical problems. For example, many common Java libraries do not ship out the box in an OSGi-friendly way. Also, the use of the thread context class loader in many libraries poses a problem for OSGi-based applications.

Solutions to these problems do exist, but they typically involve creating a more complex, more restrictive, or less familiar environment than required for traditional Java enterprise apps. Nothing comes for free. The question is whether it is worth paying the price.

9. Impala poses no extra requirements for third party library management
In terms of the management of third party libraries, Impala is no different from traditional Java applications. Third party library jars are bundled in the lib directory of a WAR file. If you change a third party library, you will need to reload your application to apply these changes.

Where Impala differs from traditional applications is the way that you manage application modules. In a WAR file, these will be bundled in a the modules directory under /WEB-INF.

The relationship between modules can be hierarchical, or even in the form of a graph of dependencies. The important point is that modularity is applied to your application's code. That is in my opinion the part which needs it most because that's the part of your application which changes frequently. If you are not sure whether you agree with this point, ask yourself the questions:

How often would you actually want to run multiple versions of the same third party library in the same application?
If you made changes to third party libraries in your application, how often would you want to apply these without restarting the application

If your answer to these questions is "not very often" or "not at all", then you probably don't need the full blown form of modularity offered by OSGi.

Some may regard the traditional approach to third party libraries as broken - but it is also very convenient when it works!

10. Impala is a practical solution for practical problems

Impala did not evolve from an ivory tower view of how Java enteprise applications are supposed to be written. Instead, it was borne around two years ago out of the need to solve practical real world problems which were harming the productivity and maintainability of a large Java project. The most notable of these were the lack of any first class modularity in traditional Spring-based application development, and the slow build/deploy/test cycles arising from unwieldy integration tests.

Impala has been designed and refactored to be simple and intuitive to work with, once you've grasped the basic concepts. It allows you to get the benefits of a modular approach to application development without many of the costs, and without being swamped by technobabble.

What are you waiting for?

Working with Impala will give you the programmer's equivalent of a "spring in your step" - you will be amazed by how easily it is to get things done.

So give it a go. Try one of the samples, read the tutorial, or kick start your own application, and let me know how you get on.

Avoiding over eager reloading in Impala

2009-09-01T17:16:00.003+01:00

The forthcoming version of Impala has new features which help to avoid "over-eager" reloading of modules, that is, module reloads which take place either unnecessarily and too frequently.

If you run an Impala application within Eclipse you can have a running application automatically reload modules to reflect changes in your workspace. You do this by adding the line


#Automatically detect changes and reload modules
auto.reload.modules=true

to the file impala-embedded.properties.

This is really nice, because it means you can redeploy your application without having to do anything, that is, no build step whatsoever.

The trouble is, redeployment can get a bit overeager. For example, some changes will automatically get picked up without the need for redeployment, for example, my Freemarker web templates. Some changes you also want Impala to ignore completely, for example, changes which occur within your subversion (.svn) directories.

The next drop of Impala now contains a mechanism to filter which files get checked for modifications, via the auto.reload.extension.includes and auto.reload.extension.excludes. An example is shown below:


#Includes only files ending with context.xml and class
auto.reload.extension.includes=context.xml,class

With the setting above, only classes and files ending in context.xml will be checked for modifications, reducing the number of spurious reloads. You can also get complete control over the timing of reloads by specifiying a touch file.


use.touch.file=true
touch.file=/touch.txt

Using a touch file removes all spurious reloads, but does require an extra build step every time in order to update the timestamp of the touch file.

Spring Faces sample project

2009-08-28T16:47:00.003+01:00

I recently put together a sample of Impala working with Spring Faces, that is, the combination of Spring Web Flow and JSF. The example is based on the Spring Faces flagship sample, a hotel booking application which uses JSF for persistence, and Spring Security for authentication/authorization.

See instructions on how to run the sample on the Impala wiki.

I have to confess that none of the technologies used in this sample are personal favourites - I'm much more inclined towards Spring MVC, Hibernate on its own, and a more straightforward templating technology like Freemarker for the view layer. That being said, it was an interesting exercise, which threw up some interesting challenges, and also important to verify that Impala could work nicely with each of these technologies without a gargantuan amount of effort on the user's part.

One of these challenges was figuring out how to set up the JSF runtime to be loaded up by the module class loader, rather than the application class loader.

More significant for the evolution of Impala itself was figuring out how to create an arbitrary mapping from the request URI path prefix (the part of the URL after the host, port and application name), and an arbitrary module. In the Spring Faces sample, I needed to map all requests with a prefix of /spring and no extension (e.g. .htm) in the path to the web module containing the Web Flow definitions and faces view. This particular requirement gave me the push I needed to implement the main features coming in the next drop of Impala, that is, the ability to map from arbitrary URI path prefixes to modules, and once within a module, to map individual requests to servlets or filters, themselves defined within the module, based on the URI extension.

The Impala Spring Faces stack is somewhat experimental, but it does offer the promise of truly modular, dynamically reloadable applications based on these technologies.

Outstanding issues
There are a few wrinkles outstanding, though. The Spring web flow long running transaction is still not working. I have added a flush() call so that the booking does persist, but this happens at the time the booking is first entered rather than confirmed. Also, it does seem possible at least in some situations to make simple changes to the flow definitions and reload these without preventing the completion of existing flows, but I'm sure there are some corner case

Impala 1.0 M6 released

2009-05-17T21:27:00.002+01:00

I am pleased to announce the 1.0M6 release of Impala. The 1.0M6 is an important release for Impala, as it is the last pre-1.0 final release to include major enhancements and API changes. I am now pretty comfortable that the APIs and abstractions are correct and suitable for the 1.0 final release, but as always welcome any feedback from users on this.

The 1.0M6 release includes a major reworking of the shared service registry and proxying mechanisms, a new Spring namespace for importing and exporting services, and enhancements to the dynamic reloading of modules.

The headline improvements include the following.

Configuration of Impala services is now much simpler, as a new Spring 'service' namespace has been provided for easily exporting and importing services.
Service export and import can now be done not only by name but also by type or by custom attributes, the latter using a model similar to that used in OSGi.
Impala's mechanism for proxying services obtained from the service registry has improved, and is now more easily configurable.
It is now possible to export and import Impala services without having to specify an interface - proxying of the service implementation class is now also supported.
Impala now supports exporting and importing services based on Spring beans which are not singletons or not created using non-singleton Factory beans. It does this in a way that is totally transparent to users of the services, effectively allowing clients to treat all beans as singletons.
Impala now provides implementations of java.util.List and java.util.Map, dynamically backed by beans imported from the service registry.

The 1.0M6 release also introduces a number of improvements to make dynamic module reloading more robust, particularly when using the module 'auto-reload' feature, as well as other minor enhancements and bug fixes.

For more information on this release see
http://code.google.com/p/impala/wiki/Release1_0M6Announcement.

Extending Spring MVC's annotation controller

2009-05-16T10:00:00.005+01:00

In my latest project I am using Spring MVC's annotation based controller. I definitely am a fan of annotations for wiring up web applications, and suppose, relatively speaking, I can claim to be an early in this area having created Strecks, an annotation based framework for Struts.

I must say I am enjoying using the new Spring MVC - it's a massive improvement over the original framework which I found pretty clunky, especially with regard to form handling.

Setup

Configuring the application is a doddle. All you need to do is register the annotation HandlerAdapter (which does the main request processing work) and HandlerMapping (which maps URLs to your controllers). You can do this using Spring config like:

<bean class="org.springframework.web.servlet.mvc.

annotation.DefaultAnnotationHandlerMapping">

<bean class="org.springframework.web.servlet.mvc.

annotation.AnnotationMethodHandlerAdapter">

and then you're ready to go. Controllers definitions can found automatically via class path scanning, or added explicitly into the Spring config files, which I prefer to do.

One really nice thing about the controllers is the simple way to map URLs to methods as well as to provide arguments to the methods, both using annotations. An example is show below:

@RequestMapping("/warehouse/postProductsSubmit.htm")
public String postProductsSubmit(
      Map model,
      @ModelAttribute("command") PostProductsForm command,
      BindingResult result) {
//do stuff

//redirect when finished
return "redirect:postProductsForm.htm";
}

So what's missing?

There were still a few bits I felt needed to be added to make the Spring MVC annotation truly usable for my application. Here's what they are.

Missing annotations for obvious argument types

The Spring MVC annotations recognise a whole bunch of argument types. Many of these will be automatically recognised from the Servlet API including HttpServletRequest, HttpServletResponse, ServletRequest, ServletResponse, HttpSession, Principal, Locale, InputStream, Reader, OutputStream and Writer. Others will be recognised from Spring MVC annotations, such as @ModelAttribute and @RequestParam (which binds a request parameter).

What would be nice would be some built in annotation types which you could extract other types of information from the Servlet API environment in an non-intrusive way. Here I am thinking of the following:

@SessionAttribute: extract and bind a named session attribute.
@RequestAttribute: do the same for a named request attribute.
@RequestHeader: extract a request header.
Plus various others

Luckily, there is an easy way of creating your own annotations for extracting information from requests and binding this to method arguments, via the WebArgumentResolver implementation. I have created a number of these in the Impala MVC extensions project. See for example the SessionAttributeArgumentResolver and it's associated @SessionAttribute annotation.

Flash Scope

Flash scope, popularised initially by Rails, is a mechanism for transferring state from one request to the next without having to pass it via URLs. It is implemented through a session scoped attribute which is removed as soon as the value is consumed in the subsequent request. It works particuarly well with redirecting after a post.

Flash scope is especially convenient for certain use cases because it combines the convenience of session-based attributes without the long running overhead of having state hanging around in a session over a long period.

Spring MVC annotations currently don't support flash scope, so added an extension to AnnotationMethodHandlerAdapter which support flash scope. Basically you can use it as follows. In your controller method, simply set a model attribute with the prefix "flash:". The attribute will be available in the next request using the @RequestAttribute annotation. An example below demonstrates this.

@RequestMapping("/submit.htm")
public String submit(
      Map model) {
//do stuff

//redirect when finished
model.put("flash:mydata", mydataObject);
return "redirect:show.htm";
}

@RequestMapping("/show.htm")
public void(@RequestAttribute("mydata") MyDataClass mydata) {
//if you redirected using flash, mydata

//will contain the mydataObject instance
//from the last call
}

No subclass hooks for manipulating model

When I first started with the annotation-based controller I found it a little frustrating that there were no subclass hooks in the provided AnnotationMethodHandlerAdapter for manipulating the model. The only way you can do this is in the mapped request method as well as in special @ModelAttribute methods, which are also present in your controllers. An example is below:

@ModelAttribute("command")
public PostProductsForm getPostProductsForm() {
return new PostProductsForm();
}

I'm not sure if this is still a problem because I have found quite acceptable workarounds which solve the problems I was trying to solve, without having to resort to such a technique. Nevertheless, is does strike me as a sensible thing to be able to do, provided it is is done in a well-defined and controlled way.

Concluding Remarks

Spring MVC annotations have added a great deal of convenience to Spring MVC without sacrificing any of the flexibility which has always been its true strength. It's not perfection, but with a few extra fairly minor features it is easy to use and work very productively with. And of course - here comes the obligatory shameless plug! - it works even better when you use it with Impala.

Why developers don't just jump at OSGi

2009-05-04T09:01:00.002+01:00

On paper, the choice to use OSGi should be an easy one. After all, OSGi offers an escape from the "jar classpath hell" that Java developers have been living with for years. The ability to compose systems through modules that can be composed and loaded dynamically promises a solutions to a range of important problems that enterprise developers have been grappling with, unsuccessfully, for years.

Yet the takeup of OSGi has been quite slow. I was curious enough the other day to take a look on Jobserve on job postings requiring OSGi, and I found only a handful of positions available. While there seems to inexorable movement towards OSGi driven in particular by application server vendors (who, remember, also drove the adoption of the now infamous EJB 1 and 2), and a few evangelists, we are yet to see a groundswell of enthusiasm from the mainstream developer community, in the same way as, for example, with technologies like Grails and, before it, Spring.

This is a shame, because I believe the ideas that underpin OSGi are fundamentally important to writing flexible systems which can remain manageable as they grow in size and complexity.

I'd like to comment on some of the reasons why OSGi has still not taken off in a way which cements it's role as the foundation for Enterprise Java applications, and also to explain why I haven't used OSGi as the basis of Impala. My intention is not to spread FUD, but to identify some of the perceptions (and potentially misconceptions) held on OSGi and to give my interpretation on to what extent they are justified.

Some people just don't "get it"
Not everybody thinks that the idea of partitioning an application into modules is a good one. Some developers are happier just to lump all classes together under a single source directory, and don't see how an application can benefit from modules. Maybe they haven't worked on projects that really require this kind of partitioning, or have suffered from a botched attempt to modularise an application. Clearly, these developers are not going to be early adopters of OSGi or, for that matter, a technology like Impala.

OSGi won't really help the productivity of my development
Clearly, there is more work involved in setting up an OSGi application than a regular Java application. You need to ensure that all your jars, both for your application and for the third party libraries, are OSGi compliant. For your application's jars, you'll be responsible for the bundle manifest yourself, making sure that its content fits in with the structure and organisation of your application. You'll definitely want some tool to make this job easier. Also, you'll have to source third party libraries which are OSGi compliant, or, in the worse case, add the necesary OSGi metadata yourself.

The productivity advantages of dynamically updatable modules will probably kick in at some point, but not until you have a smooth running development environment set up. You can accelerate this process with the help of an OSGi-based framework such as SpringSource's dm Server, or ModuleFusion.

While OSGi will undoubtedly help you write better and more flexible applications, you don't get many wild claims that OSGi will allow you to build your applications dramatically faster. Developers who come to OSGi with those kinds of expectations will probably be disappointed.

OSGi requires a complex environment
Enterprise Java has a reputation for being complex. Not only do you need to know the Java language, you need to know Spring, Hibernate, web technologies, relational databases, etc. etc.. You need to know all sorts of related technologies, test frameworks, ANT or Maven, and more. And this is just to write traditional Java applications.

To write OSGi-based Enterprise applications, there is much more to know. You'll need a good conceptual understanding of how OSGi works - both in the way that it manages class loaders and the way services are exported and consumed. Not Java 101 stuff. You'll also need a practical understanding of the idiosyncracies of your own OSGi environment. There will be differences in the way you build and deploy applications, and the way you manage the tools and runtime, depending on which OSGi containers and helper frameworks you use. You won't need to be a rocket scientist to figure this all out, but you will need some time, patience and experience. The wave of books coming out on OSGi will definitely help, but don't expect the junior members of your team to be able to jump straight into an OSGi project and hit the ground running.

How do I know it will all work?
Some people might be put off OSGi because of lingering thoughts that they will run into difficulties getting their applications to work in an OSGi environment, especially those with large existing code bases.

Some of the most commonly used frameworks out there are not very OSGi-friendly, typically either because they are designed and packaged in a not very modular way, or because they use class loaders in a way which does not align with the OSGi specification, for example, by using the thread context class loader to load classes. Naive use of these libraries in an OSGi environment will lead to unexpected problems.

You'll need to find a way to work around these issues. The hard way will be to try to do it yourself. The easy way will be to rely on a packaged OSGi solution, again such as dm Server or ModuleFusion. But remember, even here, there are trade-offs. In the case of the dm Server, you'll be very closely tied in to SpringSource as a vendor, and with ModuleFusion, you may need to accept a technology stack which does not include your favourite frameworks.

OSGi applications are difficult to test
This, in my opinion, is a real achiles heel of OSGi. Because OSGi applications need to run in an OSGi container with class loading managed in a very specific way, you cannot run low level integration tests without the tests themselves running in a container. This makes testing OSGi applications particularly challenging.

The only serious attempt I am aware of to address this problem is the Spring Dynamic Modules test framework, which dynamically creates a test bundle using your application code, launches an OSGi container, deploys the test bundle to the OSGi container (as well as the bundles you need to test plus some infrastructure bundles), and runs your test code. It's not especially pretty, but there's no substitute for real integration tests as opposed to unit tests or tests using mock objects.

For me, ease of testing is of fundamental importance in choosing technologies - it certainly is a large part of the reason for the emergence of Spring. I certainly have no appetite for a return to the days of EJB 1 and 2 when applications could only be tested on a container.

Some concluding remarks

Let me make my position clear. I am not an OSGi evangelist. I prefer to think of myself as OSGi-neutral. I have deliberately chosen not to base Impala on OSGi, but I have designed it in a way which accomodates OSGi - indeed I even have a working example of Impala running on OSGi. As OSGi gains traction - and if users demand it - Impala will provide much better support for OSGi and even offer a simple migration route to OSGi which users can choose to adopt on a per project basis.

Where are all the Groovy web frameworks?

2009-03-26T21:46:00.000Z

A little while ago when I was exploring web frameworks - as I tend to do once every few months - I spent a bit of time looking into web frameworks built in Groovy.

Groovy has all the raw materials needed for a powerful web framework implementation. It's dynamic nature and metaclass facility make it perfect for layering syntactic sugar on top of common tasks. It's very easy to dynamically update Groovy-based functionality - very little special framework is required - which makes it potentially very productive. And it already has very powerful facilities for templating, string manipulation, etc. - all capabilities required for web application development.

With all of these advantages Groovy should be a very fertile breeding ground for web frameworks. After all, for all the dozens of Java web frameworks, you could at least expect a few out there harnessing the power and convenience of Groovy.

Sadly, this does not seem to be the case. You can use Groovy to embellish plenty of existing Java web frameworks, but precious few web frameworks appear to have been built from the ground up with Groovy's power features in mind.

I'd happily be proved wrong on this point, but from my searches Grails seems to be the only real show in town.

I like Grails' web framework. It is an excellent showcase for the power of Groovy. What I don't want is the full stack Grails experience. If I were to use Grails as a web framework, I would want to embed it in an Impala-based application with a modular back-end. It was originally in the Grails 1.1 roadmap to separate the web part of Grails. However, during a recent meetup, I got it from the horse's mouth that this wasn't going to be happening any time very soon.

According to Graeme Rocher, he would consider splitting the web framework "if his users demanded it". I think he is missing the point that by not providing a separately embeddable flavour of Grails he is isolating himself from a significant untapped user base - those looking for a capable web framework but not the full Grails stack. Maybe because Grails has become so popular it is a luxury he can afford!

So where does this leave Groovy? I don't think that it is healthy for a language to be too dependent on a single framework for its popularity. The language needs - and deserves - more variety, a choice to suit a wider range of appetites.

A natural successor to Java?

2009-03-20T15:36:00.000Z

Before I started learning Java in the late 1990s, I remember picking up a book on Java and on the back cover it was described as the "natural successor to C++". Java has now reached that stage in its evolutionary cycle. It's not that Java is not not a productive and capable language. Because of the excellent tool set that you get with Java it is still more productive (particularly for me) than anything else out there, including the likes of Groovy and Ruby. But just imagine how much more productive you could be with a language which had Java's tooling support, but was free from the annoyances and deficiencies of Java, which are now generally quite well understood but very difficult to fix without breaking backward compatibility.

So what are we looking for in a language that will succeed Java? Firstly, it has to be based on the JVM. While Java as a language is running out of steam in terms of new features added, the JVM has still got real momentum, if only to be judged by the plethora of languages that run on the JVM, both from the ranks of languages which have independent lives outside of the JVM (e.g. JRuby), through to languages which are designed to work on the JVM.

Second, it still needs static typing. That's a strong personal preference - I don't see myself ditching statically typed languages altogether. Dynamic languages like Groovy are great for certain tasks - web app development, integration, etc, but for the guts of a substantial application or application framework, I really do think static typing is a must have.

Third, it needs to have substantial value add features. The one area most missing in Java is support for a more functional style of programming. As a developer who never studied computing at university, I have a background in commercially popular languages - the trend towards more functional programming styles has been around in academia for ages but is more recently moving into the enterprise programming world. While a more functional style of programming does not come that naturally to me, it's absence in Java is quite often a barrier to effective code reuse, and there is no doubt that adopting a more functional style of programming will help me to become a better programmer.

Scala seems to tick the boxes nicely in these three areas, and the fact that it is still fundamentally OO will make it more accessible to existing Java developers.

Still absorbing Scala, so its too early to say what I don't like about it. One concern is whether it is too feature rich, too complicated. Too many features may make it more powerful but may also raise barriers to adoption enough to prevent it from becoming mainstream.

Another potential contender to watch out for is Fan. I saw Stephen Colebourne's talk earlier this week, and was impressed. It strikes me as a serious attempt to fix the problematic aspects of Java without introducing any unnecessary new complexity. It has a very simple type system, and will be much more accessible than Scala, and allows easy switching from the base static typing to duck typing.

On the down side, it has a few features which may prove to be controversial, such as the inability to share mutable state between threads, and only partial support for generics. Another thing I'm note sure about is that abstracts over .NET as well as Java. This may be of interest to those who have to work in both of these environments, but I don't see much value in this for those working primarily in Java.

While these questions rumble on in the background, I intend to add support in Impala for modules implemented in different languages. Should be very simple to do this - after all, it will be mostly just a question of adding build support.

The multi-module structure of an Impala application makes development in multiple languages a good choice, with Java for core interfaces and domain classes, a language like Scala for the more complex implementation elements, and more dynamic languages suitable for modules closer to the fringe of the application.

New Impala extensions project

2009-03-09T20:53:00.005Z

I've created a new Impala Extensions project, also on Google Code. The idea is to host modules, typically extensions to Spring or Impala or both, which are generically useful, but not substantial enough to warrant their own project and too far from the core of Impala to warrant inclusion in Impala itself.

So far the extensions project contains a set of extensions to Spring annotation-based MVC framework, and also a general purpose event management framework, particularly useful for managing persistent and asynchronous events (as a lightweight alternative to JMS). I expect the list to into a number of other areas over time.

The extensions project also contains an example application which can be used as testbed for individual modules.

Roadmap Update

2009-02-12T09:15:00.002Z

I took a look at the Impala roadmap and realised that it had got a little out of date. I've corrected this - http://code.google.com/p/impala/wiki/Roadmap.

There are some pretty cool features on there way, and plenty of work to do. The good news is that a final 1.0 is not too far away.

Impala 1.0M5 released

2009-02-09T09:30:00.001Z

I am pleased to announce the release of Impala 1.0M5, which introduces many API and configuration improvements into the framework.

Following 1.0M5, only minor changes in internal APIs are now expected prior to the 1.0 final release. The 1.0M5 release also features improvements which make it much easier to configure Impala-based applications, and to add your own extensions to the framework. While Impala is still very heavily based on the Spring framework, 1.0M5 now also makes it possible to plug in other runtime frameworks into Impala's dynamic module loading mechanism.

For more information on this release see: http://code.google.com/p/impala/wiki/Release1_0M5Announcement

Impala 1.0M4 released

2008-12-17T11:31:00.003Z

I am pleased to announce the release of Impala 1.0M4, with preliminary support for OSGi.

Impala 1.0M4 now includes the ability to arrange Impala modules and their corresponding class loaders in a graph, instead of in a hierarchy as in previous releases. This feature offers application developers much more flexibility in choosing the appropriate module structures for applications.

1.0M4 is the first release of the project to include OSGi support. All Impala jars are now OSGi-compliant bundles. It is now possible to run Impala and its application modules in an OSGi container (currently Eclipse Equinox). Further improvements to Impala's support for OSGi are expected in subsequent releases.

At this stage Impala support for OSGi is still experimental. It is not recommended at this stage to use it in a production environment.

Impala promises to become the first and only project to offer a seamless transition to (and from) OSGi. There is no intention to make OSGi the default deployment model for Impala. However, OSGi is likely to become a good choice in the future for projects which will benefit from more rigorous management of third-party libraries than is possible using the traditional model.

More information on Impala's position with respect to OSGi can be found in this blog entry: http://impalablog.blogspot.com/2008/12/announcing-osgi-support-in-impala.html

For more information on this release see: http://code.google.com/p/impala/wiki/Release1_0M4Announcement

Phil Zoio

Impala Home: http://impala.googlecode.com

Announcing OSGi support in Impala

2008-12-05T09:26:00.003Z

The soon to be released version of Impala will contain OSGi support. Does this mean that I've abandoned my mission to make Impala the simplest, most lightweight, test-friendly Java dynamic module framework. Absolutely not. The idea is simply to translate the benefits of Impala into an OSGi environment.

In the future, you will no longer need to choose between OSGi and Impala. You can have the best of both worlds. As long as you have some appreciation for the value and benefits of dynamic modules in Java, then Impala can work for you, no matter what your stance on OSGi is.

If you have not looked at OSGi and have no intention of doing so, then you're already well covered by Impala. If you are likely to look at OSGi in the future, then you can take advantage now of a framework which already does most of what you'd want, and move towards OSGi when you're ready. If you're already an OSGi enthusiast, I'd encourage you to take a look now at the Impala OSGi sample. Just point your Eclipse SVN repository explorer to http://impala.googlecode.com/svn/trunk/osgi-sample, check out the contained projects, and run any of the unit tests. You'll see something like the following, for MessageServiceTest.

Note that the unit test is actually run on the Equinox OSGi container using the Spring Dynamic Modules test infrastructure. If this seems very easy, it's because it is - you won't even need to run any build scripts to get the example working!

If you're familiar with Impala you will notice that your OSGi-based application is configured in an identical manner to regular Impala applications, allowing a seamless transition from a plain Impala to an OSGi-based runtime (and back again). Impala's abstraction for representing relationships between modules provides a convenient mechanism for expressing and reloading module subgraphs, something you need to do continually during development.

Impala's OSGi support leverages much of the ground work covered by the Spring DM project. Impala also automatically generates OSGi compliant jars using Peter Kriens' BND tool when using Impala's build support, and provides a convenience target to generate an OSGi manifest so that you don't need to continually create a new set of OSGi jars each time you modify your application.

So is Impala's OSGi support production ready? Not at this point. Apart from the lack of any serious road testing, the big blocker is there is no explicit web support (yet). Additionally, Impala applications should have the option to use OSGi's service registry. I'd also like to better test framework integration so that Impala's interactive test runner and integration test suites can work equally well in an OSGi environment.

There are still some interesting technical challenges to overcome, and plenty of work to do to bring this project to its full potential. So if there is anyone out there with an interest in dynamic modules, Spring, OSGi , etc. and would like to help out, I'd love to hear from you.

So why add OSGi support to Impala?

The sweet spots in Impala are the ease in writing and running integration tests, ability to reload application modules, great web support, and a really simple, no-nonsense development environment. The sweet spots with OSGi is the way in which it gives you control over loading of third party libraries. OSGi allows you to dynamically reload third party libraries as well as concurrently run different versions of the same third party library classes, two things that you cannot do with Impala on its own.

I might stick my neck out a bit by saying that the features that I described above that are available in Impala on its own are much more valuable for the typical Java application than the extra features provided by OSGi. It is for this reason that Impala is built from the ground up to work in the simplest, most developer-friendly way possible without any reliance on third party runtime environments, tools or build systems.

That being said, there are clearly valid and potentially important use cases for the unique OSGi features. This, together with the growing industry momentum behind the technology, make OSGi support pretty much mandatory for any Java dynamic module system which demands to be taken seriously.

As I made clear at a talk in London last week, OSGi is the most powerful and sophisticated Java dynamic module system, particularly in its use of class loaders. But for many projects a simpler technology would suffice. Impala is the only dynamic module framework that offers the flexibility to choose easily between alternatives.

It is worth noting that in the upcoming version 1.0M4, Impala introduces support for expressing module dependencies via a graph rather than simply as a hierarchy. This allows individual modules to reuse functionality from multiple dependencies, rather than a single parent. This feature is particularly important both because it narrows the gap between Impala's built-in module support and that provided by OSGi, allowing for a more seamless and less constrained transition between the technologies.

Impala 1.0 M3 released with major web enhancements

2008-10-13T09:24:00.002+01:00

I am pleased to announce the release of Impala 1.0M3, with major web enhancements.

Impala is a dynamic module framework for Java enterprise application development, based on the Spring framework. With a focus on simplicity and productivity, Impala radically transforms application development using Spring and related technologies.

The Impala 1.0M3 release includes a range of improvements aimed at more robust and powerful support for web applications consisting of multiple dynamic modules. This release provides the underlying capabilities required for building dynamically reloadable web applications using frameworks other than Spring MVC, such as Struts, Wicket, Tapestry and others.

Impala 1.0M3 also includes a dynamic properties framework, as well as various other feature improvements and bug fixes. A more detailed release announcement is here: http://code.google.com/p/impala/wiki/Release1_0M3Announcement

With Impala, you can take your Spring application development to the next level without having to grapple with unfamiliar technologies or tool sets. It requires no additional third party libraries beyond those required for vanilla Spring applications. It works within existing Java deployment environments, and requires no complex runtime infrastructure or tool support.

Impala Home: http://impala.googlecode.com

Using the Thread's context class loader in a multi-module environment

2008-10-03T09:06:00.011+01:00

When you working in a multi-module environment, a problem you bump up against is how to handle the thread's context class loader.

In case you haven't seen this before, you can access the thread's context class loader using the code


Thread.currentThread().getContextClassLoader();

and you can set it using


Thread.currentThread().setContextClassLoader(classLoader);

Why it exists

So why does the context class loader exist, and how is it used? The thread context class loader was introduced in Java 1.2 to enable frameworks running in application servers to access the "right" application class loader for loading application classes.

Lets take a framework like Struts, which uses the thread's context class loader (as do many other similar frameworks). Struts allow you to declare action classes in an XML configuration file. Once the framework has read the file and figured out what class to load (at this point only identified name), the framework needs a classloader to load the class.

One naive approach to this problem would be to use the code


this.getClass().getClassLoader()

This would return the the class loader responsible for loading the framework class. There is an obvious limitation here. The framework jar would need to be packaged in a location visible only to the application class loader, and not to the system class loader or to some shared or server class loader. Again, coming back to the example of Struts, say running in Tomcat, the framework jar would need to be placed in WEB-INF/lib and not on the system class path or in a server class path such as TOMCAT_HOME/common/lib. This restriction might hold in many cases, but is not one which can be relied upon.

To solve this problem, the context class loader was invented, to give framework code a mechanism to find the "correct" class loader to load application classes. In the case of the web application, the server typically applies the web application class loader as the context class loader.

So what's the problem?

This mechanism has worked well enough until now, but is a bit of a hack, something which is certainly exposed by multi-module frameworks such as OSGi.

There is an interesting article on the Equinox wiki page on this problem and how it addresses. To save some typing (and a bit of thinking), I've picked out a quote:

Many existing java libraries are designed to run inside a container (J2EE container, Applet container etc). Such containers explicitly define execution boundaries between the various components running within the container. The container controls the execution boundaries and knows when a boundary is being crossed from one component to the next.
This level of boundary control allows a container to switch the context of a thread when a component boundary is crossed. Typically when a container detects a context switch it will set the context class loader on the thread to a class loader associated with the component which is being entered. When the component is exited then the container will switch the context class loader back to the previous context class loader.
The OSGi Framework specification does not define what the context class loader should be set to and does not define when it should be switched. Part of the problem is the Framework is not always aware of when a component boundary is crossed.

Some Solutions

Eventually, the OSGi specification will presumably come up with a standard way of addressing this problem. Until then, it is up to vendors to implement their own solutions. Equinox does so through Buddy Class Loader and Context Finder mechanisms, which you can also read about here.

Rob Harrop describes his solution here for the Spring Application Platform:

Each bundle in OSGi has it's own ClassLoader, so therefore, only one bundle can be exposed as the thread context ClassLoader at any time. This means that if a third-party library needs to see types that are distributed across multiple bundles, it isn't going to work as expected.
The Platform fixes this by creating a ClassLoader that imports all the exported packages of every module in your application. This ClassLoader is then exposed as the thread context ClassLoader, enabling third-party libraries to see all the exported types in your application.

So this will work as long as you export the types that the third party libraries needs to load.

Impala's Solution

Impala uses proxies to call across module boundaries. The proxy interceptor has access to a service reference for the target object being invoked. This service reference, as well as containing a reference to the target instance, also has a reference to the class loader responsible for loading the module which exported the target bean. Before invoking the instance, it sets the thread context loader to this class loader.

That's quite a mouthful - if code makes more sense that this English then see this souce.

Additionallly, Impala Servlets - either extensions of the Spring DispatcherServlet or those responsible for integrating with other servlets - perform a similar operation.

There are one or two gotchas. It is possible to call beans defined in parent modules without going through a proxy. In this case, the described mechanism will not be applied. I'm pretty confident though that this particular scenario won't cause any serious problems.

I'd certainly be interested in finding out over the course of time whether there are other strategies that Impala's solution won't cover.

Is the case for forking Spring building?

2008-09-22T09:34:00.007+01:00

SpringSource recently announced a new maintenance policy which I, like many in the developer community, find quite disturbing. The basic idea is that SpringSource will only publish releases to non-paying customers of SpringSource for three months after the initial release. After that, it will be up to the rest of the community to do its own releases, with community releases only coming out every so often.

Reactions to this on The Server Side and other places have been understandably very negative. Personally, I have been less than enthusiastic about some of the developments that have been coming from SpringSource since it got VC funding. See a previous entry.

This time, it has gone to far. By biting the hand that feeds it, it has risked alienating its most loyal users and putting many people off the project altogether. It also really undermines the increasing number of other community projects which are built around Spring, projects like Impala.

As the author of a number of open source projects which despite obvious (to me) technical merits never attracted a very large number of users, I know only too well that users are by far the most precious feature of an open source project, to be valued above all else.

SpringSource has not only begun to ignore it's users, it has shown the willingness to outrage them. This shows an unbelievable arrogance and complacency.

All of this is quite apart from reservations I increasingly have about the direction that SpringSource is taking Spring.

The project is no longer open
if you look at the history of SVN commits in Spring, you will notice that the project has become much less open. Around early 2005 there were plenty of commits going into Spring from individuals who are not SpringSource employees. Now commits to Spring core (the part that most people use) are only being made by SpringSource employees.

Also, strategically important projects are increasingly developed outside of the public domain - including Spring 3.0, which will be the next major release of the core framework.
We should be nervous about future licencing moves
So far the official line is that none of the none of the licences will be changed for any existing project. It would probably impossible to do this anyway. However, many new projects from SpringSource are no longer coming out with friendly open source licences (specifically ASF 2). How do we know, for example, that Spring 3.0 won't be released as a "new" project with a less friendly licence.
Obsession with OSGi
SpringSource has become obsessed with dragging the entire Java community into using OSGi. Don't get me wrong, I do like OSGi, and hope to support it with Impala, and at some point I am sure that I will be using it in real applications. But it shouldn't be at the exclusion of other things which could be done to make Spring more usable without OSGi. It's exactly for this reason that I started Impala. I wanted modules, reloadability, isolation, etc. but not necessarily with the baggage of OSGi at this point.
Key man dependencies
Every project has it's major contributors, the individuals who do most of the heavy lifting while others go around doing the talking. Spring is no different - as the commit graph shows, this is Juergen Hoeller, and it is very much down to his excellence that Spring is so well regarded. If Juergen decided to go off to pastures new, Spring core would be in a very bad place. For a project which is relied upon by so many around the world, this is not a good thing.

In short, it's hard to like the way that Spring is going. But it's not too late to change. One of two things need to happen:

SpringSource needs to take Spring back to where it came from. This doesn't mean winding up the company and going back to working for free. But it needs to reaffirm and re-demonstrate it's commitment to the spirit of open source - openness, transparency and a desire to serve it's users. This probably means lowering it's financial objectives. There's still plenty of money to be made from consulting, training, and from providing value added products and tools. But not when it contravenes the essence of what made Spring popular in the first place.
If SpringSource cannot get it's act together, the project should be forked. Because Spring runs on an ASF licence, this is quite feasible. But there are lots of question to answer. Who would be willing to do this? Would they be the right people? And where would they take the project?

Forking a project could be a dangerous move for the community, but history shows that it won't necessarily fail. Perhaps it's better for the project to take the pain now than to have death by a thousand cuts.

Judging by the strength of ill-feeling regarding SpringSource's latest announcement, I'm sure there must be other people in the wider Spring community thinking the same thing. I wonder if there are people working within SpringSource who feel the same way, or is the gravy train now just moving too fast for them to hop off?

Generic support for dynamically reloadable web frameworks

2008-09-12T20:13:00.011+01:00

One little feature that I think we're close to pulling off with Impala is the ability to embed any Servlet-based web application and have it dynamically reloadable as a module. I'm thinking of the likes of Tapestry, Struts, etc.

Up to version 1.0M2, this capability has only really existed for Spring MVC. However, to make Impala really attractive to users of other frameworks, you'd want to support these as well. After all, not every Spring user is also using Spring MVC as the web framework.

More recent changes now allow you to embed any Servlet into a the application context XML belonging to an Impala web module, using code such as:


<bean id = "delegateServlet" class="org.impalaframework.web.integration.ServletFactoryBean">
  <property name = "servletName" value = "delegateServlet"/>
  <property name = "servletClass" value = "servlet.SomeFrameworkServlet"/>
  <property name = "initParameters">
    <map>
      <entry key="controllerClassName" value = "servlet.ServletControllerDelegate"/>
    </map>
  </property>
</bean>

Note how the init parameters, the servlet name and the servlet class have been specified, as they would be for entries in web.xml.

The InternalFrameworkIntegrationServlet, whose definition is shown below, is what allows the servlet to "live" within an Impala module. It contains the glue code
that ties an invocation from outside of a module (within the context of a servlet container) to the servlet within the module.


<bean class="org.impalaframework.web.integration.InternalFrameworkIntegrationServletFactoryBean">
  <property name = "servletName" value = "myservlet"/>
  <property name = "servletClass"
    value = "org.impalaframework.web.integration.InternalFrameworkIntegrationServlet"/>
  <property name = "delegateServlet" ref = "delegateServlet"/>
</bean>

Finally, to communicate with the module itself, there is the ModuleRedirectingServlet, which simply delegates to the a servlet which is registered in the ServletContext using a name which corresponds with the name of the module (and happens to correspond with the first part of the URL's servlet path).

Here's an example configuration in web.xml:


<servlet>
   <servlet-name>module-redirector</servlet-name>
   <servlet-class>org.impalaframework.web.integration.ModuleRedirectingServlet</servlet-class>
</servlet>
<servlet-mapping>
   <servlet-name>module-redirector</servlet-name>
   <url-pattern>*.htm</url-pattern>
</servlet-mapping>

Together, these offer the capability dynamically embedding applications using arbitrary web frameworks. The idea is that you can register and load a Struts, Webwork, Tapestry or other module after the JVM has started, without impacting any existing code or without having to restart the JVM or reload the entire web application within the application server. A powerful concept.

There are definitely some gotchas to take care of. For one, it relies on object instances being loaded using the classloader obtained via Thread.currentThread().getContextClassLoader(). Also, how can we make sure that objects saved to sessions don't cause ClassCastExceptions when the module is reloaded? Also, how can we make sure that the same result does not occur because of one module attempting to access objects saved to the session by another module? And of course, we'd need to prove it working with real applications based on these frameworks, not example code assuming noddy pseudo frameworks. These kinds of problems will need to be taken care of before the problem can be considered to be fully solved. It should be fun figuring it out, though.

Impala talk at Spring User Group UK

2008-09-12T19:55:00.005+01:00

After quite a few weeks of being very busy at work, followed by a few weeks of taking it pretty easy and enjoying the summer, I'm now back to work, and trying to ramp things up a bit with Impala. On Wednesday night I did a talk at the Spring User Group UK on Impala. You can view the talk here - it's a bit grainy, but you can still hear everything. You can view the slides here.

I was given half an hour for the presentation, and I managed to squeeze it into 27 minutes, including a demo. I was pretty happy with the talk - my little example demonstrating creating a new Hibernate entity within a Spring application went without a hitch.

Hope to do a few more talks in the coming months.

Impala 1.0 M2 released

2008-07-13T22:54:00.004+01:00

Today I dropped a release of Impala 1.0M2.

It is mostly an incremental release, with a few relatively minor enhancements and bug fixes, but there is one significant feature: a mechanism for allowing module metadata to be contained within the module, instead of being externally specified, as in previous releases. I will post a bit more detail on this feature, as it does make specifying modules more intuitive and concise, as soon as time constraints ease. This change is covered in this issue report, and does involve minor backward incompatible changes.

A total of 20 issues are covered in this release. The full set are covered in this search. Enjoy.

How lightweight is Impala?

2008-05-25T09:05:00.001+01:00

In my last blog, I described a set of criteria which can be used to determine how lightweight a framework or technology is. Let's consider Impala against the criteria described above:

1. How much system resource does the technology use?
Impala adds an almost negligible overhead to a plain Spring application.

2. Do applications need special runtime support to run?
Impala simply runs on the JVM. It's only runtime dependencies are Spring and logging libraries.

3. Are any special build steps required for the application?
No, it can be run from within Eclipse without any build steps.

4. How difficult is it to test applications which use the technology?
The interactive test runner and integration test support make Impala applications easy to test in a very productive way.

5. Can applications be updated dynamically, or do updates require a restart?
Yes.

6. How easy is it to modularise applications which use the technology?
Modularity is built in from the ground up.

7. Do applications require any code generation, either source code or byte code?
None, other than that which is used in the equivalent Spring application.

8. How quick are applications to start?
Not perceptibly slower to start initially than the equivalent plain Spring application, but requiring dramatically fewer restarts.

9. How complex is the technology to understand?
Impala is simple to understand. The developer only needs to have a understanding of the basic principles, architecture and project conventions to get going. Only in very few places is interaction with Impala APIs required.

10. Is any special tool support required to be productive with the technology?
Impala works in a vanilla Eclipse installation without any additional required plugins.

Overall, I'd argue that Impala passes this lightweightness test with flying colours. I'm also pretty convinced that an equivalent comparison with alternative technologies would be less favourable.

See more details on these definitions for lightweightness.

What makes a technology lightweight?

2008-05-25T09:03:00.000+01:00

What makes a technology lightweight? Being lightweight is considered a good thing. Lightweight technologies are popular with developers because they are simpler and less cumbersome to work with. The downside, of course, is that lightweight technologies can be less feature rich than their more "heavyweight" alternatives. Whether this matters depends on whether the missing features are actually required, that is, whether the value that these features provides outweigh the overhead that is added.

Plenty of technologies are "sold" on the basis of being lightweight. While most of us have an intuitive idea of what lightweight means, it's quite useful to think of this concept in terms of a more concrete set of questions or criteria. This can be helpful in making comparisons between frameworks and technologies, and indeed for assessing claims made on behalf of particular technologies.

1. How much system resource does the technology use?
This is the most obvious measure of how heavyweight a technology is. What is the memory footprint of an application? A more narrow definition of lightweight vs heavyweight technology is most likely to focus on this aspect of the definition.

2. Do applications need special runtime support to run?
If the technology requires special runtime support, then it automatically becomes more heavyweight. A good example is EJB 2, which requires applications to be run and tested in a container. Even if the container is embeddable in a standard JVM (for example within the IDE), this can make it more difficult to test applications, particularly if tests themselves need to run within the container interactive with the code under test.

3. Are any special build steps required for the application?
If the technologies require artifacts to be packaged in a certain way, then this adds an overhead to the build cycle. A related question is, can applications be run directly in an IDE without an explicit build? The most lightweight technologies by this definition require no build steps at all - you simply fire up the application in your IDE.

4. How difficult is it to test applications which use the technology?
If a technology requires special runtime support (see 2.) or it is complex to set up dependencies for running tests, the barrier to writing effective integration tests is raised, resulting in a reduction in productivity and/or quality.

5. Can applications be updated dynamically, or do updates require a restart?
If every change made requires an application restart, this can make the technology feel more heavyweight, especially if restarts are time consuming.

6. How easy is it to modularise applications which use the technology?
Technologies which allow applications to be packaged and run in a modular way will feel more lightweight than those which require everything to be run up as an amorphous "blob", especially as the application grows in size and the need to be more fine-tuned about what is deployed becomes more obvious.

7. Do applications require any code generation, either source code or byte code?
Applications which require code generation generally involve extra build steps which can complicate the build process and/or slow down the build/test cycle. Byte code generation can hide much of this pain, but does introduce a measure of complexity which can make a technology feel heavyweight.

8. How quick are applications to start?
Technologies that are slow to start are clearly more heavyweight - they are doing more, have more going on behind the scenes, and are using more memory. They also are more likely to leave behind the perception that they are inefficient.

9. How complex is the technology to understand?
The more complex a technology is, the more heavyweight it will feel. One way of thinking of it is this: for each step you take in working with the technology, the more time you will need to spend thinking about what you are doing, relative to simply doing.

10. Is any special tool support required to be productive with the technology?
If special tool support is required to make you productive with a technology, then this technology will feel more heavyweight than one which imposes no such requirements. Now, not only do you need to worry about dependencies in your runtime environment, you need to worry about those in your tooling environment. Also, its slower to get going with the technology to start with.

You can take this methodology further by ranking frameworks and technologies on a scale of 1 to 10 on each of these points. Some frameworks and technologies which purport to be lightweight are actually considerably less so when evaluated against these criteria.

In my next blog, I will evaluate Impala's lightweightness against these criteria.

JavaWUG BOF 37: talk on Impala Framework

2008-05-21T20:21:00.005+01:00

Just a quick update on my talk at the Java Web User Group in London last night. Thanks for those who made - as always it was an enjoyable evening.

Here's a copy of the presentation.

A main focus of my presentation was a demo based on an Impala version of the Spring Petclinic application - one of the main reference applications bundled with the Spring Framework.

During the demo I introduced a new feature to the application, involving a new database table, some new Hibernate mappings, persistence code, as well as some enhancements to the web presentation layer. The changes were made without having to restart the application during the demo, all updates were made on the fly.

The source code for the demo is available from the Impala subversion repository. The changes made during the demo are in this file.

Talk next Tuesday on Impala at JavaWUG

2008-05-16T17:04:00.005+01:00

Next Tuesday evening I am doing a talk on Impala in London. Here are the details:

http://www.jroller.com/javawug/entry/javawug_bof_37_impala_framework

You can register for the event here:
http://skillsmatter.com/event/java-jee/javawug-bof-37

The talk is at 6:30pm at Skills Matter in London. I'll be explaining why I created Impala, how it can dramatically accelerate Spring-based development and allow you to create more modular, maintainable applications. The format will a mixture of slides and demos.

Impala 1.0M1 released - first public release

2008-05-10T21:17:00.003+01:00

I am pleased to announce the first public release of Impala, a dynamic module framework for Java enterprise application development.

Impala 1.0 M1 can be downloaded from http://code.google.com/p/impala/downloads/list

Impala builds on the Spring Framework to provide a genuinely modular, highly productive environment for web-based applications. It allows you to divide Spring-based applications into a hierarchy of modules which can be dynamically added, updated or removed to an existing running application.

Impala's modularity features make it possible to write applications which are much easier to maintain than plain Spring applications. Impala enables applications which can grow very large without exploding in complexity. Impala also enables genuine productivity enhancements over plain Spring development, through the dynamic module loading capability, seamless integration with Eclipse, and the efficient test management features. Impala also features basic built-in build support, based on ANT, and dependency management capabilities.

With Impala, you can take your Spring application development to the next level without having to grapple with any unfamiliar underlying technologies or tool sets. It requires no additional third party libraries beyond those required for plain Spring applications. It works within existing Java deployment environments, and requires no complex runtime infrastructure or tool support.

For more information on getting started with Impala, see http://code.google.com/p/impala/wiki/GettingStarted.

Spring Application Platform validates Impala's existence

2008-05-05T22:10:00.003+01:00

A couple of days ago I wrote a blog in which I was somewhat negative about the Spring Application Platform. My issue is partly with the problems SpringSource is trying to solve, and also the way that they are trying to solve them.

The problems being targeted are twofold: first, the ability to get more precise control over which versions of which class are loaded by which libraries in the JVM, and secondly, the need for modular applications.

What I've been saying all along is that while the ability to get precise control over class versioning is worthwhile, it also involves quite a lot of pain, and for most developers, this pain will not be worthwhile.

What is really needed is modularity. SpringSource have gone straight for OSGi to achieve both modularity and class versioning. The problem is that this solution also brings is a lot of complexity. So in order to manage this complexity, an extra solution is needed to hide this complexity. Enter the Spring Application platform.

The problem with all of this is that there seems to be an implicit assumption that effective, dynamic modularity is not possible without OSGi. How about another approach? Why not try to see how far you can get with just the standard Java class loading mechanisms? Spare yourself all the hassle where it's not needed, and go straight for the sweet spot. That's the Impala approach. OSGi may come to Impala too, but it won't be forced on users.

Surprisingly, you can get pretty far without OSGi. For one, you can have multiple versions of your own applications loaded simultaneously in different modules. You can do on the fly updates. You can also get tremendous productivity advantages from simple and lightweight mechanisms for managing integration tests, with fewer restrictions on your runtime environment than you will have with OSGi.

What you don't get without OSGi is the ability to have multiple versions of third party libraries loaded concurrently within the same JVM. The question you have to ask yourself is this: how important is this feature to you? How much pain are you prepared to go to achieve it? After all, hundreds and thousands of Java applications have been written that don't rely on this feature. In my time as a Java developer, I can only think of at most a handful of occasions where class versioning issues have presented an issue. It does not seem like the kind of problem that deserves a solution with a far reaching impact on your technology choice, runtime environment and tool set.

In short, I am apparently in agreement with SpringSource that modularity and dynamic reloading are two features whose absence has been a real limitation to development using the Spring Framework. This certainly convinces me that the last year or so that I have spent working on Impala has not been a waste of time. Indeed, the benefits experience by projects I have worked on using Impala suggest the contrary.

Spring Application Platform - is it a big mistake?

2008-05-02T18:15:00.009+01:00

It was with some consternation that I read the Server Side thread on Spring's announcement of a new application platform.

Firstly, the Spring Application Platform, as it's called, is essentially a Spring OSGi integration on steroids. I've voiced concerns on how well OSGi is going to be received by ordinary developers, as against application server vendors. I'm yet to be persuaded that regular
Java developers will be particularly enthusiastic about dealing with the idiosyncracies of OSGi. There is more than a hint of EJB history about it. SpringSource (and application server vendors?) have identified OSGi as the way that we ordinary mortals must now follow.

Spring Application Platform (and Spring Dynamic Modules) are great if you've already bought into OSGi. As a general platform for enterprise application development - at this point I'm yet to be convinced.

Secondly, with its Application Platform, SpringSource has moved squarely into the application server market. This is something which I didn't expect them to do, and I don't think it's a very good idea. It's too invasive, too big a step in one particular direction for me to feel comfortable with. I think of the Spring Framework as a technology which works seamlessly with a broad variety of platforms and application servers. With the Application Platform, they're pinning their colours to a particular mast - a technology stack based on Equinox and Tomcat, and setting themselves in competition with other application servers.

Thirdly, there is the question of the license. For the first time, SpringSource is introducing a major product which is not based on the Apache V2.0 license, instead going for a GPL license. That's a major change that reflects a real difference in the way that the company is positioning itself in the marketplace.

Finally, the question in my mind is where the Spring Framework itself fits into all of this. I've always thought of the Spring Framework as the flagship product coming from the Rod Johnson crew. It didn't matter if SpringSource came up with one or two turkeys in its portfolio, because at least the core framework is solid. But with the Spring Application Framework, they're betting big. They've clearly had some of their best brains working on the project for some time. A failure won't be quite as easy for the development community to brush off.

So what will happen now? Will the Spring Framework become the "poor relation" of the Application Platform? Will new features and improvements go into the Spring Framework, or will this suffer in the future at the expense of the Application Platform. The waters have definitely been muddied.

The Spring Application Platform is the biggest announcement to come out of the Spring team for some time. It also looks like it could be a big mistake. Spring became popular in the first place as a practical, community driven solution to the real problems with Java enterprise applications, with a focus on simplicity. The latest offering seems to be moving in a rather different direction.

Thoughts on integration testing

2008-04-30T19:07:00.007+01:00

Everybody (pretty much) knows what they mean by unit testing, and on the whole, I would expect most people to have a broadly speaking common understanding of what this means. Unit testing of course involves testing the behaviour of a particular unit of code (e.g. Java class) in isolation.

When it comes to integration testing, it's probably not quite as clear cut. For example, does a test need to cover an entire use case to be considered an integration test? What about a database test which only covers a single JDBC-based method? Is this an integration test?

For me, the key point of distinction is the realness of the test. An integration test is such because it embodies the real behaviour of the system. The more a test relies on simulating aspects of a system's behaviour, the less real it is, and further it moves away from being an integration test.

This brings me on to the question of what is an integration test in a Spring/Impala environment. Here, an integration test will typically always involve using a real Spring application context rather than constructing the fixtures programmatically. It will involve involve connecting to a real database, rather than an in-memory test database (e.g. HSQLDB). It will make limited if any use of mock objects. All in all, integration tests in a Spring environment are closer in behaviour to that exhibited by the real system.

Another question worth addressing is this: what is the relative value of integration tests versus unit tests? A working set of integration tests gives me much more confidence than a set of unit tests which are not backed by integration tests.

One difficulty with integration tests is that they are much less direct than testing using unit tests. It's harder to establish the precise link between the tests and what is actually getting tested. On the plus side, you can get a lot of test coverage with very little code through integration tests.

The other problem with integration tests is, of course, they can be painful to write. Specifying dependencies required for integration tests can be fiddly and time consuming. Also, integration tests can be slow to run. For these reasons, some developers shy away from integration tests, preferring to rely on unit tests plus manual testing of use cases using the deployed application. The big cost is that quality of the testing regime in these circumstances is much diminished, making it much easier for bugs to slip through.

With Impala, integration tests are so easy to write and quick to run, that there really are no excuses. Integration tests are easy to write because dependencies are expressed at a high level through composition of modules. They are quick to run because of the dynamic module loading support used by the interactive test runner, and the efficient way that suites of unit tests are managed within Eclipse. See this wiki page for more details.

Impala and Strecks

2008-04-12T14:59:00.007+01:00

A few of you may about my involvement in another open source project, Strecks, which was a set of Java 5 specific extensions to Struts. Yes, remember Struts, the one that used to be so popular a few years ago. There are a few similarities between Impala and Strecks, but also a few differences, that I would like to comment on.

First, the similarities. Just like Strecks was an extension to Struts, Impala is an extension to Spring. I wrote Streck to address what I perceived to be limitations of Struts. These limitations of Struts are now fairly well understood. In a similar way, I wrote Impala to address shortcomings of Spring - in particular, the lack of first class support for modules.

Strecks allowed me to write web applications with Struts the way I believed they should be written. The changes I made with Impala have allowed me to write Spring applications in the way that I would like to write them. Some of the real headaches that came with attempting to scale Spring applications, attempting to build in sophisticated configuration options, have just gone away.

Now to the differences. The timing of Strecks was unfortunate. Just as I was in the reasonably advanced early stages of development, an announcement was made that Struts was to merge with Webwork, with future Struts 2 development using the Webwork code base. That was the death knell to Struts as we knew it. I believed at the time that Strecks was a genuine way forward for Struts, without having to throw away the code base and stick a finger at the huge Struts user community. However, once the decision had been made, any project based on old Struts had little chance of gaining any long term traction. I still took the project to a 1.0 final release, but there seemed little point in actively developing the project from then onwards, unless it served my own needs directly. Since then, I haven't been working on Struts projects more than intermittently, so such a need hasn't arisen in any real sense. That being said, it's still getting 100 to 200 downloads a month, so it hasn't disappeared completely off the map. Oddly, I am now spending a bit more time on a Struts-based project, so it might be a good opportunity to revive Strecks and put in a couple of features I had wanted to add. Also, I might focus on Strecks as a target for web framework integration with Impala.

In contrast to Struts, I don't think there is any chance that the Spring code base is going to be deprecated any time soon. It's an extremely healthy project, arguably more so than any other in the Java community. Secondly, internally, Spring is well architected. It has all the extension points I have needed for Impala to fit naturally into the existing design paradigm. I have never felt the need to correct Spring's faults. Struts, on the other hand, had flaws in its architecture that were not always easy to get around when implementing features in Strecks.

My hope is that the Spring community will share my view that Impala elegantly solves some of the real practical problems which come when developing large, complex Spring-based applications, delivering substantial productivity benefits at the same time. It has been a fascinating and enjoyable project to work on. It would be nice to see some of these benefits being shared more widely. But first I need to get the public release out!

More about Impala scaffolding

2008-04-04T14:12:00.005+01:00

It is important to appreciate the developers are busy people - if they are going to spend an hour looking at your project, then this has to be an hour well spent. For this reason, there is a real focus in Impala on making the developer experience as seamless as possible. Everything should work out the box. It's only when you start doing interesting stuff, stuff specific to your problem domain, that you should have to do any real work.

In my previous post I described the simple scaffolding system that is available for Impala. This is not supposed to compete with the scaffolding provided by Ruby on Rails, Grails and such projects. There is a fundamental difference in approach. Impala's scaffolding is only supposed to take you to the point where you have a clean slate to work on. It is not a code generation framework, and I have no intention of moving into that space with it. However, getting to a clean slate position - where you can actually start working on your application - is not a zero work task for many projects. For a project such as Impala which is designed to support very complex applications, you want to make this as simple as possible. I don't want developers who take time out to try Impala to spend their first hour creating directory structures, setting up class paths, and downloading third party libraries from various sources.

The scaffolding I've created with Impala is designed to help you transition to square one as quickly as possible. Once you get there you have a working Impala Hello World application, with working module definitions, a working web application, interactive integration tests and a JUnit suite. With a decent internet connection, running through the scaffolding steps should only take a few minutes - allowing time to play, try things out and do some fun stuff.

Scaffolding for Impala

2008-04-01T19:56:00.015+01:00

It's been a little while since my last post, and since then, Impala has taken a few steps forward in the background. One of these is a simple scaffolding mechanism, which allows you to go from an empty empty Eclipse workspace to a working application in just a few simple steps.

First, start by downloading the latest snapshot distribution of Impala from the subversion repository.

http://impala.googlecode.com/svn/trunk/impala/impala/dist/impala-SNAPSHOT.zip

You can do this via the web browser. From Linux or Mac OSX you may find it more convenient to use curl.

cd ~
curl -o impala-SNAPSHOT.zip http://impala.googlecode.com/svn/trunk/impala/impala/dist/impala-SNAPSHOT.zip

You can then unzip the Impala distribution:

unzip impala-SNAPSHOT.zip

Once you've unzipped Impala, set the IMPALA_HOME environment property.

IMPALA_HOME=~/impala-SNAPSHOT
export IMPALA_HOME

In windows, you will probably want to use the GUI to do the same.

Now, change to IMPALA_HOME, and run the following command:

ant -f scaffold-build.xml scaffold:create -Dimpala.home=./

You will then be guided through an interactive process where you will need to specify the following information:

the name of the project containing the root Impala module. This project is a kind of a master project, and will also be the project from which you will typically run ANT build scripts when this is necessary.
the name of the project containing a non-root module. In a real world application, non-root modules would contain implementations of DAOs, service methods - anything really. In a substantial real world application there will be several if not many non-root modules, together forming a hierarchy of modules.
the name of a project containing a web module. Although it is possible to have multiple web modules in a single application, the simple scaffolding starts with just one web project.
the name of the repository project. This contains the third party jars used by the different application modules.
finally, the name of the test project. The test project is really a convenience from which you can easily run suites of tests for the entire application.

The last time I ran this command, the following output was produced:


ant -f scaffold-build.xml scaffold:create -Dimpala.home=./
Buildfile: scaffold-build.xml

scaffold:input-workspace-root:
[input] Please enter name of workspace root directory:
/Users/philzoio/workspaces/scaffold

scaffold:input-main-project:
[input] Please enter main project name, to be used for root module:
main

scaffold:input-module-project:
[input] Please enter name of first non-root module:
module

scaffold:input-web-project:
[input] Please enter name of web module:
web

scaffold:input-test-project:
[input] Please enter name of tests project:
test

scaffold:input-repository-project:
[input] Please enter name of repository project:
repository

scaffold:create-confirm:
[echo] Workspace root location: /Users/philzoio/workspaces/scaffold
[echo] Main (root) project name: main
[echo] First non-root module project name: module
[echo] Web project name: web
[echo] Tests project name: test
[echo] Repository project name: repository
[input] Press return key to continue, or CTRL + C to quit ...


scaffold:create:
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold
[copy] Copying 6 files to /Users/philzoio/workspaces/scaffold/main
[copy] Copied 6 empty directories to 5 empty directories under /Users/philzoio/workspaces/scaffold/main
[copy] Copying 3 files to /Users/philzoio/workspaces/scaffold/main
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/main/spring
[copy] Copying 4 files to /Users/philzoio/workspaces/scaffold/module
[copy] Copied 5 empty directories to 4 empty directories under /Users/philzoio/workspaces/scaffold/module
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/module
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/module/spring
[copy] Copying 11 files to /Users/philzoio/workspaces/scaffold/web
[copy] Copied 8 empty directories to 4 empty directories under /Users/philzoio/workspaces/scaffold/web
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/web
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/web/spring
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/test
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/test
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/repository
[copy] Copied 2 empty directories to 1 empty directory under /Users/philzoio/workspaces/scaffold/repository

BUILD SUCCESSFUL

Before we import the Eclipse project, there are just two more steps to follow:

First, go the main newly project, and run the following two commands:

cd /Users/philzoio/workspaces/scaffold/main
ant fetch
ant get

The fetch command will copy the Impala libraries into the repository project of the new workspace, as shown by the following output.


ant fetch
Buildfile: build.xml
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

repository:fetch-impala-from-lib:
[copy] Copying 12 files to /Users/philzoio/workspaces/scaffold/repository/main

repository:fetch-impala-from-repository:

repository:fetch-impala:

fetch:

BUILD SUCCESSFUL

The get command downloads the necessary third party libraries, as defined using a simple format in dependencies.txt files.


ant get
Buildfile: build.xml
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

shared:get:
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

download:get:
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold/repository/build
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold/repository/test
[get] Getting: http://ibiblio.org/pub/packages/maven2/commons-logging/commons-logging/1.1/commons-logging-1.1.jar
[get] To: /Users/philzoio/workspaces/scaffold/repository/main/commons-logging-1.1.jar
[get] Getting: http://ibiblio.org/pub/packages/maven2/commons-logging/commons-logging/1.1/commons-logging-1.1-sources.jar
[get] To: /Users/philzoio/workspaces/scaffold/repository/main/commons-logging-1.1-sources.jar

...

[download] ******************************************************
[download]
[download]                RESULTS OF DOWNLOAD OPERATION
[download]
[download] org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2.jar resolved from
http://ibiblio.org/pub/packages/maven2/org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2.jar
[download] org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2-sources.jar resolved from
http://ibiblio.org/pub/packages/maven2/org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2-sources.jar
[download]
[download] ******************************************************

get:

BUILD SUCCESSFUL
Total time: 3 minutes 6 seconds

We're now ready to import our projects into Eclipse. Start by opening Eclipse in the newly created workspace.

Use the menus File -> Import ... -> General -> Existing Projects Into Workspace. When prompted, set the import base directory to the workspace root directory. This should bring up a dialog box as shown below.

Select all of the projects and import them.

If you reach this point and no errors are showing in your workspace, the congratulations! You have just set up a new Impala workspace.

Let's test it out:

Running up the web application

Using CTRL-Shift + T, find the class StartServer. Right click, then select Run As ... Java Application.

This will start a Jetty Server and run up a server on port 8080.

The text shown on the console view of Eclipse will look something like this:


2008-04-01 20:56:10.408::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2008-04-01 20:56:10.479::INFO:  jetty-6.1.1
2008-04-01 20:56:10.963:/web:INFO:  Initializing Spring root WebApplicationContext
INFO : BaseImpalaContextLoader - Loading bootstrap context from locations [META-INF/impala-bootstrap.xml, META-INF/impala-web-bootstrap.xml, META-INF/impala-jmx-bootstrap.xml, META-INF/impala-web-listener-bootstrap.xml]
INFO : ScheduledModuleChangeMonitor - Starting org.impalaframework.module.monitor.ScheduledModuleChangeMonitorBean with fixed delay of 2 and interval of 10
INFO : LoadTransitionProcessor - Loading definition root-module
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module root-module: [file [/Users/philzoio/workspaces/scaffold/main/bin]]
INFO : LoadTransitionProcessor - Loading definition module
INFO : ModuleContributionPostProcessor - Contributing bean messageService from module module
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module module: [file [/Users/philzoio/workspaces/scaffold/module/bin]]
INFO : LoadTransitionProcessor - Loading definition web
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module web: [file [/Users/philzoio/workspaces/scaffold/web/bin]]
2008-04-01 20:56:12.133:/web:INFO:  Initializing Spring FrameworkServlet 'web'
INFO : ExternalLoadingImpalaServlet - FrameworkServlet 'web': initialization started
INFO : ExternalLoadingImpalaServlet - FrameworkServlet 'web': initialization completed in 21 ms
2008-04-01 20:56:12.168::INFO:  Started SelectChannelConnector @ 0.0.0.0:8080
DEBUG : ScheduledModuleChangeMonitor - Completed check for modified modules. No modified module contents found
DEBUG : ScheduledModuleChangeMonitor - Completed check for modified modules. No modified module contents found

You can connect to the server using the URL:

http://localhost:8080/web/message.htm

Note that Impala is started up to automatically detect changes in your modules, and reload modules in response to these changes. You can play with this mechanism by making changes to classes such as MessageController (in the web project) and MessageServiceImpl (in the module project).

Running the standalone interactive client

Use Eclipse to find the JUnit test class MessageIntegrationTest. Again, run this as a Java application, and execute tests, reload modules etc, using the interactive test runner. Here's some example output:


log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
Test class set to test.MessageIntegrationTest
Unable to load module corresponding with directory name [not set]
Starting inactivity checker with maximum inactivity of 600 seconds
--------------------

Please enter your command text
>test
No module loaded for current directory: main
Running test testIntegration
.Hello World!

Time: 0.055

OK (1 test)


Please enter your command text
>reload
Module 'root-module' loaded in 0.096 seconds
Used memory: 1.5MB
Max available memory: 63.6MB


Please enter your command text
>module module
Module 'module' loaded in 0.035 seconds
Used memory: 2.1MB
Max available memory: 63.6MB


Please enter your command text
>t
No module loaded for current directory: main
Running test testIntegration
.Hello World!

Time: 0.012

OK (1 test)

Run the suite of tests

Any of the JUnit integration tests can be run as a regular unit test in Eclipse, with green bar and all. From the tests project, find the class AllTests. This contains a suite of tests covering all the tests in the project. Run this as a regular unit test, and you will see the following:

There's plenty more to get your teeth stuck into, but having a working web application, a working suite of tests, and interactive tests which can be run out the box is a helpful start.

Impala working on Tomcat as war

2008-02-05T20:12:00.000Z

Today I reached a significant milestone with Impala. I successfully deployed and ran the Impala as a war file with reloadable modules, in Tomcat. Modules are placed as jars in the folder /WEB-INF/modules.

This is a big step forward. Until now, Impala has been developed and deployed in Eclipse or using ANT and an embedded Jetty runner. That's great for development, but for production, a war file is much more convenient - just having a single file which can be dropped into a web container is much more deployment friendly than having to set up a special folder on the target machine which has a structure which mirrors the development environment.

A war deployment option was one of the main features which stands in the way of a public release. There's quite a bit of tidying up to do in the way that the build/packaging process works, but it's fair to say that we're now a big step closer to a first public release of Impala.

Modularity: what is it, and why is it important?

2008-01-30T20:47:00.000Z

Modularity is taken for granted as something which has value in software engineering, perhaps without always a well articulated understanding of what it is, and why it is valuable. I came to this realisation when I found myself having in some difficulty attempting to explain why the modularity that Impala brings can be beneficial to a large application. Instinctively, I felt like I had a good understanding, but it's useful to put this understanding into words.

What is modularity?

I googled remarkably few articles dedicated to the subject. The definition in the wikipedia entry was quite good, from which I quote the relevant sections:

"Programs that have many direct interrelationships between any two random parts of the program code are less modular (more tightly coupled) than programs where those relationships occur mainly at well-defined interfaces between modules.
Modules provide a separation between interface and implementation. A module interface expresses the elements that are provided and required by the module. The elements defined in the interface are visible to other modules. The implementation contains the working code that corresponds to the elements declared in the interface."

Modularity in Impala

This definition fits in very well with how modularity is implemented in Impala. Modules fit into a hierarchy. An Impala module only depends directly on its parent. This means that two modules at the same level have no direct dependencies on each other. Modules communicate with each other through well defined interfaces - specifically Java interfaces defined in a shared parent module.

Child modules dynamically contribute their implementation to a proxy bean defined in a higher level. I'm also planning contributions to a service registry, as is the case with OSGi. The proxies can themselves be dynamically registered, although static wirings of these are necessary to support the traditional named bean style dependency injection.

Why is Modularity Important?

Modularity is a key weapon in reducing with complexity in a large project. In the same way that blocks of functionality within a component should be partitioned into classes according to responsibilities, so should parts of an application be partitioned into modules. Like classes in an OO model, each module should have a well defined - albeit higher level - set of responsibilities. Ideally, these responsibilities should be closely related to each other.

The benefits of modularity, in my view, are as follows:

by definition, it reduces the unnecessary cross dependencies between code for different parts of a system. This makes code easier to maintain and extend. Impala enforces this kind of modularity because classes at the same level of the module hierarchy are not visible to each other.
it's easier to specify the constituents of a modular system at a higher level. An Impala application is just a collection of modules. In practical terms, it is much more convenient to specify an Impala application as a small number of modules than as a large number of Spring configuration files.
it's easier to read and understand a modular system. Each module has a clear and relatively narrowly defined set of functions, which can be more easily documented and explained than in the case of a system which is not modular.
a dynamic modular system such as Impala also has the advantage that parts of the system can be separately added, upgraded and removed. This is not possible in a system which is not modular.

Without modularity, a large system rapidly degenerates into one which is extremely hard to maintain. I would go so far as to argue that a large system which is not modular is doomed to excessive complexity, maintenance issues, and an unnecessarily high bug rate.

Spring Exchange 2008 in London

2008-01-16T19:49:00.000Z

Today I attended the Spring exchange, held in London. Not only is it good to get away from the day job occasionally, but Spring is obviously the critical base technology for Impala, so an opportunity to get a quick, thorough and free update from the horse's mouth, as it were, is very welcome. Thanks to Spring Source and organisers Skills Matter.

Rod Johnson delivered the keynote, which as usual found another interesting angle to his perennial topic - the success of Spring. This time, the theme was the "The Changing of the Guard". It was all about the disruption that was taking place in enterprise development as a whole, with the leading technologies of yesteryear, J2EE and .NET, with their "One Size Fits All" approach, increasingly viewed as inadequate. A bleak future awaits J2EE, is his view. It's hard to disagree. Spring of course has pride of place in this future, evidenced for example by the continual upward rise of "Spring and Java" job adverts. As you would expect, he had some interesting ideas and perspectives. But I did feel like his talk went on a bit long ... I'm told it was 59 slides long!

The second talker was Sam Brannen, who went through the new features of Spring 2.5, fairly succinctly although without too much fanfare. Spring 2.5 now targets Java 1.6, fully supports JEE 5, and ships OSGi compliant bundles. However, the biggest areas of development in Spring 2.5 core are the use of annotations, in particular the myriad of ways they can be used to set up Spring bean definitions, effectively reduce the amount of configuration code that you need to write, but not without a few health warnings having thrown in.

Some of the annotation mechanisms include:

JSR 250 common annotations, such as @PostConstruct, @PreDestroy and @Resource
the Spring @Autowire annotation
@Component, which identifies a class as a Spring bean

plus a whole bunch of others. While I am certainly a fan of annotations for specialised uses, such as in web frameworks such as Strecks, there is certainly the danger that such a sudden and vast proliferation of annotations will be confusing to many users. Personally, for most bread and butter bean definitions I think I will carry on preferring to use the basic XML definitions, with namespace support where appropriate.

On the whole, the annotations and new namespace implementations are really syntactic sugar which don't really tackle a real problem which remains with Spring: the absence of modularity as a first class concept at the heart of the framework. SpringSource aims to address this using OSGi, and I'm told that Spring OSGi will ship 1.0 next week. Nevertheless, Spring OSGi will not quite get to the crux of the problem, because a solution is needed now, and OSGi still needs to go through plenty of growing pains before it reaches maturity in the enterprise space.

The third talk before lunch was Dave Syer, who described the improvements to Spring's web offering. This is one area where they really have made impressive strides, and there is plenty more to come. Admittedly, it is also an area where some real catch up was necessary.

The bit I liked best was Spring @MVC, the new Annotation-based controllers, which combined the component scanning, offer a much neater way to wire up web applications. Couldn't help thinking back to my work on Strecks, and the similarity in the approach! JSF support is much improved, but I'm not terribly bothered about this. There's more AJAX'y stuff, and interestingly, gracefully degrading versions of the JSF components.

Both the afternoon sessions involved the impressive Adrian Colyer and Rob Harrop double act. One of the nice things about going to these kinds of conferences is you discover some unexpected things: this time, how they were using Eclipse MyLyn to group together different files used for different parts of their demos. A useful pedagogical technique. I liked Adrian's presentation on the internals of the Spring runtime, with some well crafted diagrams, and Rob had some good perspectives on how to use Spring in the field.

Something which really struck a chord for me was his advice on the structure of projects for large, complex applications, stressing the importance of modularity, and suggesting some practices for selecting between different configurations for development, testing and production. Yes, I'm back to my favourite themes: modularity and configurability, and of course, Impala. Not only does Impala bring modularity to Spring applications, but has some a couple of neat ways of tweaking module configurations without requiring any nasty build hacks.

Unfortunately, I didn't get to the final round table Q&A session due to home commitments, but if I had, I would have asked about Spring 3.0. There was surprisingly little spoken about Spring 3.0, and the plans they seem to have are evolutionary, not dramatic, at least as far as the core project is concerned.

A good day out, and I didn't even miss my train coming home!

Spring users: seven reasons why you'll like Impala

2008-01-12T21:17:00.000Z

Here are seven reasons why you'll like Impala if you like already Spring:

Spring encourages interface based programming. Impala takes this one step further: modules communicate with each other using interfaces, and interface implementations are contained within modules.
Modules help you to organise your code, and eliminate unnecessary cross-code and cross-bean dependencies. Modules provide a much simpler mechanism for identifying high level relationships between parts of an application than is possible using raw application context definition files and relationships between individual beans.

What's not to like? There's virtually nothing that you can do in Spring that you can't do in Impala in exactly the same way as you would do it in vanilla Spring. You can still use virtually any of the Spring APIs, any Spring feature, technique, configuration mechanism, etc., unmodified! There's no new API or language to learn. You don't need to throw away any best practices. If anything, Impala makes it easier to enforce best practices, because it strongly encourages modular applications with well defined interfaces (based on Java interfaces).
A small learning curve. To start benefiting from Impala, all you need to do is organise you project according to a well defined set of conventions, and to add a bean definition into the relevant Spring context files. Otherwise, everything is the same.

Impala is not asking you to embrace a new programming model, set of APIs or even language. Simply leverage your Java and Spring knowledge, but to greater effect.
Spring is great for test driven development in that it helps you manage dependencies better. Impala takes this a step further. The interactive test runner makes writing integration tests - still a pain point with plain Spring development - really easy, hence encouraging test driven development at every level.
Tasks such as reloading configurations, updating application logic, refreshing logging configurations are simple with Impala. Impala leverages Spring's first class JMX support to simplify these tasks.
Impala defines a convention-based project structure which makes sense both in terms of productivity and enterprise Java development best practice.

Spring's Petclinic Sample, Impala style

2008-01-07T21:23:00.001Z

I've spent a bit of time converting the Spring Petclinic sample so that it runs with Impala. Petclinic it is actually a showcase for the various Spring data access technologies. I've cut down the Impala Petclinic sample to work with following configuration:

data access using Hibernate
MySQL database
simple service and web tiers

To run the application, follow these steps:

svn co http://impala.googlecode.com/svn/trunk/petclinic petclinic
Open Eclipse, with the workspace set to the checkout directory (petclinic)

Now set up the database:

First, from petclinic/db, run createDB.txt (as root). This creates the petclinic user
mysql -u root -p < createDB.txt
Then as the petclinic user, insert the tables
mysql -u petclinic -ppetclinic petclinic < initDB.txt
To insert data, run
mysql -u petclinic -ppetclinic petclinic < populateDB.txt

To run the tests from Eclipse, simply run the main class AllTests as a JUnit test, which is in the project petclinic-tests.

To run up the web application, simply run the main class StartServer, which is in the project petclinic-web. This time, run it as a (main) Java application.

To run up the interactive test runner, run HibernateClinicTest as a Java application. Type u for usage. You can run this class as a standard JUnit test - it's part of the AllTests suite.

Next on my list is to create a sample based on Spring's Petstore sample (originally Clinton Begin's JPetstore application, a sample with a rather long history). This is more interesting, as it is a larger application with a greater variety of components, so should be an interesting showcase for Impala.

When will Impala be "go public"?

2008-01-06T13:51:00.000Z

At the moment, there is still no public release of Impala. There have been no announcements on any popular web sites, to the Spring mailing lists or forums, or to any other public forum. This is deliberate. This is not because the necessary features are not present. Most of the functionality needed for a public release is present, and the code is usable and good quality. There is is enough functionality to provide real benefit to a project using Impala.

What still remains is to be absolutely sure that the Impala internal interfaces are correct. For the last few months (yes, literally!), I have been doing extensive refactoring, with the result that I am progressing towards this point. If users come up with a whole new set of requirements and ideas on how it can be used, there should be a clearly defined set of interfaces to build upon.

The first public release of Impala won't be just before going 1.0. However, it will only occur when I have reached the point that I am satisfied that the interfaces are as correct as I can make them, to support extensibility and maintainability moving forward. I think I am quite close. The interfaces are in pretty good working order, and the organisation of the code seems to make good sense.

I need to spend a bit more time working on the interactive test runner, and on deployment issues. Most of the work I've been doing with Impala has been within Eclipse itself. And of course, samples and documentation, although the first public release doesn't need to depend on these.

Impala and OSGi

2007-12-17T21:38:00.000Z

The project most closely related to Impala is Spring OSGi, so it's tempting to try make some comparisons on the two projects, or at least on their approaches to solving the problems they are tackling. I should preface this by saying that I am talking from a certain position of relative ignorance: my understanding of Spring and OSGi is based mostly on what I have read in the documentation and a small amount of playing with samples some time ago. I've since been to a talk on OSGi at JavaWUG in London, and read a fair portion of the OSGi spec. I hope this has given me sufficient understanding to make a few remarks which reflect my impressions of the differences between Impala and the world of Spring OSGi.

Spring OSGi and Impala are tackling an overlapping set of problems

What Impala and Spring OSGi have in common is that they are both dynamic module based systems.

Impala provides a developer productivity solution through build support, an interactive test runner, and support for efficient, fast running integration tests.

OSGi, on the other hand, tackles a wider set of problems relating to visibility of classes. OSGi allows you to run multiple versions of third party libraries within the same JVM, which can prevent problems which may occur from different libraries depending on incompatible versions of the same third party library. You can think of OSGi as defining multiple class spaces for third party libraries as well as application code.

Impala makes a clear distinction between application code and third party libraries. There is only a single class space for third party libraries. If there is a clash between third party libraries, then you are no better off (or worse off, for that matter) than you would be in a standard Java/pre-OSGi world. You would still need to resort to whatever workaround would apply in that case.

While third party library clashes are certainly a theoretical possibility, it is quite rare in my experience to actually be a victim of such problems in a way which couldn't be addressed relatively easily through a simple workaround. Others may have different experiences in this regard. However, it does seem that there is quite a bit of overhead required to make sure that the Jars you use fit in nicely with OSGi requirements, so that they "play nicely" in OSGi world. It's the kind of overhead that I think most developers would look for ways to avoid for as long as they are able to do.

Impala has more of a focus on developer productivity

The purpose of Spring OSGi seems primarily to bring the benefits of OSGi to Java, via Spring. In that sense, it is OSGi centric. The fundamental trade-off with OSGi is one of productivity. Are you prepared to go to quite a lot more trouble in defining quite precisely the nature of dependencies between libraries, with the benefit that you will never get ClassCastExceptions due to classloader issues, and that you will be able to dynamically update library versions. Make no mistake, these benefits come at a cost. One very experienced OSGi developer described working with library dependencies OSGi as a "pain in the arse", something you wouldn't want to do without tool support.

Impala, by contrast, does not impose these kinds of constraints. In fact, it's very essence is about productivity. Dynamic module reloading can benefit productivity, because it means that integration tests can be written on the fly against a running application.

Impala Support for OSGi

Impala has all the internal interfaces necessary for seamlessly supporting OSGi, probably via Spring OSGi. I'm definitely considering adding this in the future. For the time being, that's not on the immediate horizon - getting the project into a publicly releasable form is a higher priority right now.

Impala being used on a "real" project

2007-11-25T21:52:00.000Z

I'm pleased to say that I am now using Impala for what looks like may and should become a core and major development at the company where I am working. The results have been great. Progress has been more rapid than I have expected, and a lot of this is down to the capabilities provided by Impala.

I haven't run into any major obstacles that couldn't be overcome with relatively little effort. Dynamic reloading is working well. Test execution speed - a bugbear of the previous version - is quick. There are now around 200 tests, of which around 30 or so are integration tests which load up various bits of the application. The application currently consists of 21 Hibernate entities. Total execution time of all the tests - integration included- is about six to eight seconds for initial startup, plus the same again to run the tests.

It's a major moment for any open source project is when it gets its first real commercial user. In this case, the actual user is still primarily myself. That's a good thing - it means that on a day-by-day basis I am getting to validate the features and functionality, discover bugs when the appear, and get ideas for improvements and new features.

Successful open source projects are often extractions from existing successful commercial projects. Others are developed from a vision of the technology the developer would like to work with on his or her next project. Impala started off this way. The danger for the latter kind of projects is that the opportunities to use the technology in anger on real commercial projects may not present themselves that readily, with the danger that you don't get the chance to "eat your own dog food". At the moment, I am eating my own dog food by the spadeful, and it tastes good!

New package and domain names

2007-11-06T22:01:00.001Z

I've been spending a bit of time working on the package layout of Impala classes. While the project is still at an early stage, it makes sense to try to get this right.

The first thing to do was to change the root package name. I've had to do this after moving the code from Java.net to Google code. The Impala root package is now org.impalaframework. It would have been nice to use org.impala, but I'm politely sticking to the convention that you should own the domain that you use (yes, I have actually bought the domain org.impalaframework!). I certainly didn't want to use com.googlecode; if I need to move the project again, then I don't want to have to do another root package rename. org.impalaframework is a bit wordy, but it at least there is a precedent in the form of the root package names used for Spring: org.springframework. Hope this doesn't make me too much of a copycat.

All this being said, the convention that you should to use a package name which corresponds to a domain name that you own kinda sucks. Of course, this is only a convention, and you can choose to ignore it. I may choose to do so, but for now, I'm more interested in getting the structure of the packages and classes right. That means getting the names of classes and packages right, getting their locations correct and eliminating package cycles. This process should settle down soon. It will need to before I'm ready to do the first public release.

Grails Exchange

2007-10-22T20:47:00.000+01:00

Last month I did a talk on behalf of the Java Web User Group (JavaWUG) at the Grails Exchange. Unfortunately, I did not go to the full conference but it was nice to pop in and attend a couple of the evening sessions, do my bit, and join in for a beer and chat aftewards.

The thrust of my talk was on embedding Grails into Java applications. Grails is a full-stack framework, which aims to solve an overlapping set of problems to Impala. I feel lured towards Groovy, but every time I get put off by my perception that Groovy is too slow. I'm not talking about execution speed, more compile speed.

Anyway, there could be a nice synergy with Impala and Grails - Impala for a standard Spring-based back end, and Grails for the front end. That's certainly a part of the reason for my interest in embedding Grails in a typical Java application.

Impala, of course, works out of the box with Spring MVC, but it would be great to get integration going with other frameworks. One of the key requirements for a web framework integration with Impala is that the web application is itself dynamically reloadable. Of course, you could integrate Impala with any statically reloading web application, but it seems a bit pointless having dynamically reloadable back-end functionality while still having to restart the application every time you want to make a change to one of your presentation classes. Grails seems to fit the dynamic reloadable requirement quite well. However, unpicking it's functionality so that you can reuse it outside of the Grails environment is not trivial.

Here's some code which goes part of the way to achieving this task.


public static void main(String[] args) throws IOException {
    MetaClassRegistry registry = GroovySystem.getMetaClassRegistry();

    if (!(registry.getMetaClassCreationHandler()
        instanceof ExpandoMetaClassCreationHandle))
        registry.setMetaClassCreationHandle(new ExpandoMetaClassCreationHandle());

    FileSystemXmlApplicationContext cpc =
        new FileSystemXmlApplicationContext("web-app/WEB-INF/grails-context.xml");
    DefaultRuntimeSpringConfiguration springConfig =
        new DefaultRuntimeSpringConfiguration(cpc);

    GrailsApplication application = (GrailsApplication) cpc.getBean("grailsApplication");
    application.registerArtefactHandler(new ControllerArtefactHandler());
    application.registerArtefactHandler(new TagLibArtefactHandler());
    application.registerArtefactHandler(new DomainClassArtefactHandler());

    application.getConfig().put("plugin.includes", "filter, controllers, taglib");

    DefaultGrailsPluginManager pluginManager =
        new DefaultGrailsPluginManager(new Class[] {}, application);
   
    WebApplicationContext context = springConfig.getUnrefreshedApplicationContext();
    pluginManager.setParentApplicationContext(context);
    pluginManager.loadPlugins();

    pluginManager.doPostProcessing(context);
    pluginManager.doRuntimeConfiguration(springConfig);
    pluginManager.setApplicationContext(context);
    pluginManager.doDynamicMethods();
   
    WebApplicationContext ctx = springConfig.getApplicationContext();

    System.out.println(Arrays.toString(ctx.getBeanDefinitionNames()));
}

Note that this code will fire up Grails, load its web-specific plugins. The trick now is to get the rest of the environment working, so that a full Grails web application can slot neatly into Impala's environment. Something I'll be trying in the next few weeks.

Impala moved to Google code

2007-10-04T19:37:00.000+01:00

I've just moved Impala from Java.net to Google code.

Here's the new link: http://impala.googlecode.com/

Here's why.

I originally set Impala up with Java.net because I wanted project hosting which would be simpler, quicker and easier to maintain than Sourceforge, where I had previously hosted projects. On the whole things did work out that way, but there are a few things about Java.net that I don't like.

First, and this is a biggie, there doesn't appear to be anonymous SVN source access. This means that people who want to check out the source code need to sign up to Java.net, and for some people this is definitely likely to be a barrier to trying out the project, and hence to the project's adoptions.

On a related issue, there doesn't seem to be direct web-based SVN browsing available. I can't send a link which points directly to a source file on the SVN repository.

Second, it seems to have quite a complicated roles and permission system that I didn't immediately understand and still don't. Frankly, the only roles or permissions I'd be interested in are project owner, committers and everyone else. Anyone should be able to see anything; only developers should be able to change things.

Thirdly, while everything seems nicely packaged, the tools seem a bit dated. The bug tracker, which I hadn't started looking, looked like a wrapper around bugzilla.

On the whole, the site has a bit of a clunky feel - I'm not that big a fan of the CollabNet software. I wouldn't expect this to change dramatically too soon, because I can just imagine the pain that would be involved with upgrading.

Finally, setting up the project took a bit long. It took a couple of weeks before it was put in the Enterprise Incubator category. There's just a bit too much process around the project setup. I want plain and simple hosting. The users and community as a whole, and not some Sun employees, will decide if there is any merit in the project.

Google Code, on the other hand, was really no frills in its setup. The tools seem very simple to use. Everything is laid out clearly. I didn't need to spend ages figuring out how the site works. Google generally does a good job at creating usable applications, and this one doesn't seem to be any different.

Of course, there is anonymous SVN access for casual users. I like the tabbed layout of the project home page. The bug tracker looks simple but very functional, and the same goes for the wiki. And, as you'd expect with Google, there is good indexing and search capability, so that documentation and source code is searchable.

All in all, it made sense to change sooner rather than later, as the project is in a very early, pre-adoption phase. The job would be much more painful later on. Now I've got to rename the packages!

2007-09-22T19:21:00.000+01:00

I've added some timing information for running Impala in interactive test mode, and the numbers are quite interesting.

I start by firing up a JUnit test class, called WineDAOTest, which looks like the following:


public class WineDAOTest extends BaseDataTest {

    public static void main(String[] args) {
        PluginTestRunner.run(WineDAOTest.class);
    }

    public void testDAO() {

        WineDAO dao = DynamicContextHolder.getBean(this, "wineDAO", WineDAO.class);

        Wine wine = new Wine();
        wine.setColor("red");
        wine.setVineyard("Chateau X");
        wine.setTitle("Cabernet");
        wine.setVintage(1996);
        dao.save(wine);

        Collection winesOfVintage = dao.getWinesOfVintage(1996);
        System.out.println("Wines of vintage 1996: " + winesOfVintage.size());
        assertEquals(1, winesOfVintage.size());

        wine.setVintage(2000);
        wine.setColor("rose");
        dao.update(wine);

        Wine updated = dao.findById(wine.getId());
        assertEquals(2000, updated.getVintage());

    }

    public PluginSpec getPluginSpec() {
        return new PluginSpec("parent-context.xml", new String[] { "dao", "hibernate", "merchant" });
    }

}

The Impala features have been added by

exposing a main method which calls PluginTestRunner
obtaining the Spring context bean using DynamicContextHolder.getBean(...)
providing an implementation of getPluginSpec(), which tells the PluginTestRunner which plugins to include as part of the test. Note that we have plugins for the basic Hibernate configuration, for the DAO implementations, and for the service classes (the merchant plugin)

We now run the class in interactive mode by running it as a Java application in Eclipse (instead of running it as a JUnit test. Here's the output.

--------------------------------
Enter u to show usage
>u
l [testClass] to load test class
[testName] to run test
reload [plugin name] to reload plugin
reload to reload parent context
s to show test methods
r to rerun last command
r to rerun last run test
e to exit
--------------------------------
Enter u to show usage
>s
Available test methods:
testDAO
--------------------------------
Enter u to show usage
>testDAO
Running test tests.WineDAOTest
.Wines of vintage 1996: 1
Time: 3.264
OK (1 test)

--------------------------------
Enter u to show usage
>reload
Parent context loaded in 0.551 seconds
Used memory: 3.4MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload
Parent context loaded in 0.34 seconds
Used memory: 4.2MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload
Parent context loaded in 0.521 seconds
Used memory: 3.8MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload hibernate
Plugin hibernate loaded in 0.21 seconds
Used memory: 4.3MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload merchant
Plugin merchant loaded in 0.16 seconds
Used memory: 4.6MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload dao
Plugin dao loaded in 0.141 seconds
Used memory: 4.5MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload
Parent context loaded in 0.511 seconds
Used memory: 4.1MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>reload
Parent context loaded in 0.43 seconds
Used memory: 3.9MB
Max available memory: 63.6MB

--------------------------------
Enter u to show usage
>testDAO
Running test tests.WineDAOTest
.Wines of vintage 1996: 1
Time: 0.081
OK (1 test)

--------------------------------
Enter u to show usage
>testDAO
Running test tests.WineDAOTest
.Wines of vintage 1996: 1
Time: 0.07
OK (1 test)

--------------------------------
Enter u to show usage
>

Notice how the initial test run took 3.2 seconds, which is how long we'd expect a small Spring/Hibernate test run to take. After that it gets more interesting.

Subsequent reloads of the application context (shown by the reload call) take only a fraction of the time: 0.55 seconds, 0.34 seconds, 0.51 seconds and 0.34 seconds. This is only a fraction of the time it takes to load the application context the first time.

Reloading individual plugins in much quicker: hibernate took 0.21 seconds, dao took 0.14 and merchant took 0.16 seconds. This means, at least for a small application, we can reflect changes almost instantly. Even for an application which loads 10 times more slowly, the numbers are still quite acceptable.

Running the test without doing the reload is also extremely fast: compare 0.07 and 0.08 seconds with the original 3.2 seconds.

The Rationale of Micro Hot Deployment

2007-09-19T18:49:00.000+01:00

A big part of Impala's reason for being is that it supports micro hot deployment of Java applications. So what is micro hot deployment, and why is it a valuable concept?

Traditional hot deployment in Java is all about being able to update an application on the fly, typically a WAR or EAR on an application server. In reality, this kind of hot deployment is not terribly useful. Firstly, it often comes with memory leaks, which mean that after a couple of redeployments you may end up running out of memory. Secondly, because it is a coarse-grained redeployment, it can take quite a long time. For example, if the server itself only takes two or three seconds to start up, and the application takes 20 to 30 seconds to load, then in terms of downtime you little worse off doing the server restart.

Micro hot deployment involves hot (re)deployment of parts of an application. Java Servlet containers such as Tomcat already support this in a very limited sense. For example, JSPs, which are compiled into Servlets, can typically be updated without an application restart. This is because it is safe to associate a JSP with its own class loader, because the class which the JSP compiles to it will never be referenced by another application class. It is very much at the end of the application dependency chain.

The only other form of hot deployment considered safe by application servers is redeployment of full applications. This is a fairly brute force tactic for getting around the limitations and pitfalls of Java classloaders.

Other technologies do a much better job of implementing micro hot deployment. The likes of PHP and Ruby on Rails, for example. Even within the Java camp, scripting based solutions such as Grails, based on Groovy, have tackled this problem head-on.

Unfortunately, Java frameworks have been pretty slow to follow suit. Tapestry 5 now promises that application classes will be reloadable on the fly in production, not just development, and Wicket has a reloading filter which can be used to hot-redeploy pages. But these are the exception, not the rule, and most Java frameworks are less ambitious in this department.

Solving the hot redeployment is something that Java application frameworks need to get right if they want developers to stay with the platform in the long term. This means working with classloaders, which can be a tricky business. But tricky does not mean impossible.

Impala tackles the problem of micro micro hot deployment for Spring-based applications. It allows for the division of the application into modules which can be reloaded separately. One of the important principals that it recognises is that in terms of frequency of change, not all application artifacts are created equal. Let's start with the most frequently changed parts of an application:

configuration flags: application specific properties which allow for switchable behaviour of the system at runtime. A trivial example would be a flag testMode which would be switched off in production.
UI templates: without any changes to the structure of the application, changing these can change the way the application appears.
infrastructure configuration: here we're talking about resources such as database connection pools, which existing independent from any application classes.
business rules: parts of the application which carry out the business logic of the application. These can, for example, be changed without having to change the domain model of the application.
domain model: we're moving closer to the root of the dependency graph here - changes to domain model objects typically can have downstream effects on all of the items listed above.
shared utility classes: these are units of code which are shared by different parts of the application, that don't relate directly to the domain model or business processes of the application. Since they haven't been packaged into separate third party libraries, they are technically still part of the application.
third party libraries: these tend to change much less often than the artifacts of the application itself.

Impala recognises the different life cycles of the different types of artifacts within an application. For example:

it is possible to reload the core of the application (domain model plus shared utility classes) and all of the business components without reloading any of the third party classes. This is important, because one of the things that takes Java apps so long to start is the need to load classes from third party libraries. Typically, the number of third party classes used, directly or indirectly, is much larger than the number of application classes.
it is possible to reload one of the business components without reloading the application core or any other business components
it will be possible to reload infrastructure configurations without reloading the core of the application, and vice versa. Note that the latter is possible because infrastructure components don't depend directly on the core application classes.
it is possible to reload tests without having to reload any of the application classes they are testing. This can dramatically cut down the time to write integration tests.
configuration flags and UI templates are less of a challenge to reload dynamically. The former can be done via reloadable configuration files, while the latter is usually supported by good web frameworks or servlet containers.

Micro hot deployment is about finding ways to minimise the granularity of artifact reloading, so that only artifacts that have changed, and those which depend on them, have to reload when changes are made. The benefits for improving developer productivity are obvious, and their are important potential benefits for live updates of in deployment environment, too.

Impala's non-Maven approach to simpler builds

2007-09-14T10:55:00.000+01:00

One of the goals of Impala is to have a pure work out-the-box feeling. If you download the distribution, you should be able to set up a new project with just one or two commands. Once you've done this, the project structure should be ready for you. The build infrastructure should be just there. As long as you obey the project structure conventions, then you should be able to plug in an existing build system into your new project.

All of this is behind the ideas that drive Maven. Maven defines a standardised folder locations, and an existing build infrastructure which you can just plug in and use. All of these ideas are great, but I don't want the project to depend on Maven. I'm still trying to decide whether I should make the project structure conventions conform to those of Maven. The advantage is that Maven users would be able to simply Mavenize their project by adding a POM xml. The disadvantage is personal - I don't particularly like the Maven project structure conventions. I wouldn't have chosen them for myself.

Right now I'm pretty close to having a pure out-of-the-box ANT based build system ready for Impala. It's taken quite a lot of time, but it's starting to feel much more right. Basically, the build will be enabled using the following combination:

a build.xml in the project root directory. The build.xml needs to have a property called impala.home, which defines where the Impala install files have been dumped to on the file system
a set of other project-specific properties which need to be specified, either within the build.xml itself a properties file
a set of imports of build scripts sitting in the impala.home folder

Here's an example:


<?xml version="1.0"?>
<project name="Build" basedir=".">

  <property name = "workspace.root" location = "..">
  <property name = "impala.home" location = "${workspace.root}/../impala">
  <property file = "build.properties">
  <import file = "${impala.home}/project-build.xml">
  <import file = "${impala.home}/download-build.xml">

</project>

Note that the clean, compile, jar and test targets typical in a build system are found in project-build.xml. This file in turn relies on your project structure conventions to find the resources it needs. Similarly, adding download-build.xml adds support for obtaining dependencies, for example from a Maven ibiblio repository. You can make this build file the master build file for a multi-project build, simply by adding an import to shared-build.xml and adding the project.list property, as shown in this example:

<?xml version="1.0"?>
<project name="Build" basedir=".">

   <property name = "workspace.root" location = ".."/>
   <property name = "impala.home" location = "${workspace.root}/../impala"/>

   <echo>Project using workspace.root: ${workspace.root}</echo>
   <echo>Project using impala home: ${impala.home}</echo>
 
   <property file = "build.properties"/>
   <import file = "${impala.home}/project-build.xml"/>
   <import file = "${impala.home}/shared-build.xml"/>
   <import file = "${impala.home}/download-build.xml"/>
   <import file = "${impala.home}/repository-build.xml"/>
 
   <target name = "get" depends = "shared:get"/>
   <target name = "fetch" depends = "repository:fetch-impala"/>
   <target name = "clean" depends = "shared:clean"/>
   <target name = "dist" depends = "shared:all-no-test"/>
   <target name = "test" depends = "shared:test"/>
 
</project>

with the extra property in build.properties:

project.list=wineorder,wineorder-hibernate,\
   wineorder-dao,wineorder-merchant,wineorder-web

I'm looking forward to getting all of this work done, so I can get back to what Impala is really supposed to be doing, which is supporting dynamic Spring modules. But all of this functionality is terribly important for making the whole experience as painless as possible for end users.

Simple dependency management, Impala style

2007-09-13T10:14:00.000+01:00

I've been spending a bit of time working on a simple dependency management system for Impala. You're probably asking: why don't you just use something like Maven or Ivy? Aren't they supposed to be doing that job for you?

Well, the problem is that I've decided I don't want transitive dependency management. What I do want is a simple way for users of the project (including myself, of course) can easily find and download the dependencies they do need. In a funny way, Maven does come to the rescue. It defines a standard format for repository jar entries, which goes like this:

[base url]/[organisation name]/artifact name]/[version]/[artifact name]-[version].jar

For example, I can hold of the Spring framework jar into using the URL
http://ibiblio.org/pub/packages/maven2/org/springframework/spring/2.0.6/spring-2.0.6.jar

Thankfully, Maven also defines a standard format for source jars.

[base url]/[organisation name]/artifact name]/[version]/[artifact name]-[version]-sources.jar

Okay, so I don't want transitive dependencies, then what do I want?

an easy reliable way to get hold of dependencies without having to go to each and every individual project web sites
source jars that match the downloaded binaries, so that I can easily attach source for debugging
a simple way of saying where the downloaded files should go

Impala uses the concept of a project-specific repository. A project specific repository will have a simple structure consisting of parent folders and subfolders as shown below:

Now, all I need is a simple mechanism which says how individual artifacts are added to this repository. The mechanism Impala uses is a simple text file, which has entries such as below:


main from commons-logging:commons-logging:1.1
main from commons-io:commons-io:1.3
main from log4j:log4j:1.2.13
main from org.springframework:spring:2.0.6 source=true
main from cglib:cglib-nodep:2.1_3
test from junit:junit:3.8.1
test from org.easymock:easymock:2.2
test from org.easymock:easymockclassextension:2.2

For each artifact, I say optionally whether I want to include source (overriding whatever the default setting happens to be).

I still need to do the work of figuring out what the dependencies are, but once I've got there, life is pretty peachy. You get the best of both worlds - simplicity and control.

I still need to do some tweaks to get the mechanism fully ship shape, but it's basically working pretty well. Another feature is that you can specify multiple source locations, including your local Maven repository, so if the artifact you need happens to be lurking on your file system from a previous download, you don't need to waste time trying to get it from the net.

Impala, ANT and transitive dependencies

2007-09-05T20:08:00.000+01:00

Any project needs a build environment. For a project like Impala, which is trying to provide a simpler environment for Spring development, the choice of build tool is important.

ANT has been around for an awful long time, but it's not exactly the most sexy software around these days. For a while it was the only show in town. Nobody likes writing build scripts, and the one thing that ANT does make you do is write build scripts for your projects.

These days, the obvious other choice is Maven. Maven's two selling points are thus: it saves you from having to write build scripts, and it is supposed to handle dependency management for you. Of course, it cannot do everything. You need to tell Maven what libraries you want to use. Maven in turn will find a whole bunch of other libraries that you don't necessarily know about, which the libraries that you do want to use themselves depend on. This is called transitive dependency management. The problem, though, is twofold. Maven relies on dependency information put in POM XML files. This information is only as good as the person editing the POM. It's not foolproof. Secondly, even if the POM is accurate, it may be more complete than it needs to be. How does it deal with optional dependencies? If these are included in the POM, but not needed for the parts of the library that you are using, then you get a bloated repository. And the one thing I can't stand is bloat.

There are other things I don't like about Maven. Frankly, it tries to do too much, without necessarily doing anything very well. Lots of people I speak to have complained about this. It entails a lack of control over your build environment, which scares me. The horror stories you hear about don't seem to happen with ANT. It's a pain, but one way or another, you know you are going to be able to get the job done.

A second option is Gant, based on Groovy. Gant is built on ANT too, but uses a Groovy DSL to make ANT scripting easier. It seems much better for complex scripting, because you can use proper language features such as iteration and conditionals much more easily than with ANT. Two problems with Gant, though. There's more setup involved, because it requires Groovy and Gant to be installed (as well as ANT, perhaps). Also, it's a bit slow. Groovy is much slower to compile than Java, which has a noticeable impact on build times.

For these reasons, I'm back to ANT. If I need to do anything complex, like handle dependency management, or iterate through projects, I'll write a custom task for that. It's easy enough. ANT is not pretty, but it is effective.