Spring Application Platform - is it a big mistake?

It was with some consternation that I read the Server Side thread on Spring's announcement of a new application platform.

Firstly, the Spring Application Platform, as it's called, is essentially a Spring OSGi integration on steroids. I've voiced concerns on how well OSGi is going to be received by ordinary developers, as against application server vendors. I'm yet to be persuaded that regular
Java developers will be particularly enthusiastic about dealing with the idiosyncracies of OSGi. There is more than a hint of EJB history about it. SpringSource (and application server vendors?) have identified OSGi as the way that we ordinary mortals must now follow.

Spring Application Platform (and Spring Dynamic Modules) are great if you've already bought into OSGi. As a general platform for enterprise application development - at this point I'm yet to be convinced.

Secondly, with its Application Platform, SpringSource has moved squarely into the application server market. This is something which I didn't expect them to do, and I don't think it's a very good idea. It's too invasive, too big a step in one particular direction for me to feel comfortable with. I think of the Spring Framework as a technology which works seamlessly with a broad variety of platforms and application servers. With the Application Platform, they're pinning their colours to a particular mast - a technology stack based on Equinox and Tomcat, and setting themselves in competition with other application servers.

Thirdly, there is the question of the license. For the first time, SpringSource is introducing a major product which is not based on the Apache V2.0 license, instead going for a GPL license. That's a major change that reflects a real difference in the way that the company is positioning itself in the marketplace.

Finally, the question in my mind is where the Spring Framework itself fits into all of this. I've always thought of the Spring Framework as the flagship product coming from the Rod Johnson crew. It didn't matter if SpringSource came up with one or two turkeys in its portfolio, because at least the core framework is solid. But with the Spring Application Framework, they're betting big. They've clearly had some of their best brains working on the project for some time. A failure won't be quite as easy for the development community to brush off.

So what will happen now? Will the Spring Framework become the "poor relation" of the Application Platform? Will new features and improvements go into the Spring Framework, or will this suffer in the future at the expense of the Application Platform. The waters have definitely been muddied.

The Spring Application Platform is the biggest announcement to come out of the Spring team for some time. It also looks like it could be a big mistake. Spring became popular in the first place as a practical, community driven solution to the real problems with Java enterprise applications, with a focus on simplicity. The latest offering seems to be moving in a rather different direction.

Thoughts on integration testing

Everybody (pretty much) knows what they mean by unit testing, and on the whole, I would expect most people to have a broadly speaking common understanding of what this means. Unit testing of course involves testing the behaviour of a particular unit of code (e.g. Java class) in isolation.

When it comes to integration testing, it's probably not quite as clear cut. For example, does a test need to cover an entire use case to be considered an integration test? What about a database test which only covers a single JDBC-based method? Is this an integration test?

For me, the key point of distinction is the realness of the test. An integration test is such because it embodies the real behaviour of the system. The more a test relies on simulating aspects of a system's behaviour, the less real it is, and further it moves away from being an integration test.

This brings me on to the question of what is an integration test in a Spring/Impala environment. Here, an integration test will typically always involve using a real Spring application context rather than constructing the fixtures programmatically. It will involve involve connecting to a real database, rather than an in-memory test database (e.g. HSQLDB). It will make limited if any use of mock objects. All in all, integration tests in a Spring environment are closer in behaviour to that exhibited by the real system.

Another question worth addressing is this: what is the relative value of integration tests versus unit tests? A working set of integration tests gives me much more confidence than a set of unit tests which are not backed by integration tests.

One difficulty with integration tests is that they are much less direct than testing using unit tests. It's harder to establish the precise link between the tests and what is actually getting tested. On the plus side, you can get a lot of test coverage with very little code through integration tests.

The other problem with integration tests is, of course, they can be painful to write. Specifying dependencies required for integration tests can be fiddly and time consuming. Also, integration tests can be slow to run. For these reasons, some developers shy away from integration tests, preferring to rely on unit tests plus manual testing of use cases using the deployed application. The big cost is that quality of the testing regime in these circumstances is much diminished, making it much easier for bugs to slip through.

With Impala, integration tests are so easy to write and quick to run, that there really are no excuses. Integration tests are easy to write because dependencies are expressed at a high level through composition of modules. They are quick to run because of the dynamic module loading support used by the interactive test runner, and the efficient way that suites of unit tests are managed within Eclipse. See this wiki page for more details.

Impala and Strecks

A few of you may about my involvement in another open source project, Strecks, which was a set of Java 5 specific extensions to Struts. Yes, remember Struts, the one that used to be so popular a few years ago. There are a few similarities between Impala and Strecks, but also a few differences, that I would like to comment on.

First, the similarities. Just like Strecks was an extension to Struts, Impala is an extension to Spring. I wrote Streck to address what I perceived to be limitations of Struts. These limitations of Struts are now fairly well understood. In a similar way, I wrote Impala to address shortcomings of Spring - in particular, the lack of first class support for modules.

Strecks allowed me to write web applications with Struts the way I believed they should be written. The changes I made with Impala have allowed me to write Spring applications in the way that I would like to write them. Some of the real headaches that came with attempting to scale Spring applications, attempting to build in sophisticated configuration options, have just gone away.

Now to the differences. The timing of Strecks was unfortunate. Just as I was in the reasonably advanced early stages of development, an announcement was made that Struts was to merge with Webwork, with future Struts 2 development using the Webwork code base. That was the death knell to Struts as we knew it. I believed at the time that Strecks was a genuine way forward for Struts, without having to throw away the code base and stick a finger at the huge Struts user community. However, once the decision had been made, any project based on old Struts had little chance of gaining any long term traction. I still took the project to a 1.0 final release, but there seemed little point in actively developing the project from then onwards, unless it served my own needs directly. Since then, I haven't been working on Struts projects more than intermittently, so such a need hasn't arisen in any real sense. That being said, it's still getting 100 to 200 downloads a month, so it hasn't disappeared completely off the map. Oddly, I am now spending a bit more time on a Struts-based project, so it might be a good opportunity to revive Strecks and put in a couple of features I had wanted to add. Also, I might focus on Strecks as a target for web framework integration with Impala.

In contrast to Struts, I don't think there is any chance that the Spring code base is going to be deprecated any time soon. It's an extremely healthy project, arguably more so than any other in the Java community. Secondly, internally, Spring is well architected. It has all the extension points I have needed for Impala to fit naturally into the existing design paradigm. I have never felt the need to correct Spring's faults. Struts, on the other hand, had flaws in its architecture that were not always easy to get around when implementing features in Strecks.

My hope is that the Spring community will share my view that Impala elegantly solves some of the real practical problems which come when developing large, complex Spring-based applications, delivering substantial productivity benefits at the same time. It has been a fascinating and enjoyable project to work on. It would be nice to see some of these benefits being shared more widely. But first I need to get the public release out!

More about Impala scaffolding

It is important to appreciate the developers are busy people - if they are going to spend an hour looking at your project, then this has to be an hour well spent. For this reason, there is a real focus in Impala on making the developer experience as seamless as possible. Everything should work out the box. It's only when you start doing interesting stuff, stuff specific to your problem domain, that you should have to do any real work.

In my previous post I described the simple scaffolding system that is available for Impala. This is not supposed to compete with the scaffolding provided by Ruby on Rails, Grails and such projects. There is a fundamental difference in approach. Impala's scaffolding is only supposed to take you to the point where you have a clean slate to work on. It is not a code generation framework, and I have no intention of moving into that space with it. However, getting to a clean slate position - where you can actually start working on your application - is not a zero work task for many projects. For a project such as Impala which is designed to support very complex applications, you want to make this as simple as possible. I don't want developers who take time out to try Impala to spend their first hour creating directory structures, setting up class paths, and downloading third party libraries from various sources.

The scaffolding I've created with Impala is designed to help you transition to square one as quickly as possible. Once you get there you have a working Impala Hello World application, with working module definitions, a working web application, interactive integration tests and a JUnit suite. With a decent internet connection, running through the scaffolding steps should only take a few minutes - allowing time to play, try things out and do some fun stuff.

Scaffolding for Impala

It's been a little while since my last post, and since then, Impala has taken a few steps forward in the background. One of these is a simple scaffolding mechanism, which allows you to go from an empty empty Eclipse workspace to a working application in just a few simple steps.

First, start by downloading the latest snapshot distribution of Impala from the subversion repository.

http://impala.googlecode.com/svn/trunk/impala/impala/dist/impala-SNAPSHOT.zip

You can do this via the web browser. From Linux or Mac OSX you may find it more convenient to use curl.

cd ~
curl -o impala-SNAPSHOT.zip http://impala.googlecode.com/svn/trunk/impala/impala/dist/impala-SNAPSHOT.zip

You can then unzip the Impala distribution:

unzip impala-SNAPSHOT.zip

Once you've unzipped Impala, set the IMPALA_HOME environment property.

IMPALA_HOME=~/impala-SNAPSHOT
export IMPALA_HOME

In windows, you will probably want to use the GUI to do the same.

Now, change to IMPALA_HOME, and run the following command:

ant -f scaffold-build.xml scaffold:create -Dimpala.home=./

You will then be guided through an interactive process where you will need to specify the following information:

• the name of the project containing the root Impala module. This project is a kind of a master project, and will also be the project from which you will typically run ANT build scripts when this is necessary.
• the name of the project containing a non-root module. In a real world application, non-root modules would contain implementations of DAOs, service methods - anything really. In a substantial real world application there will be several if not many non-root modules, together forming a hierarchy of modules.
• the name of a project containing a web module. Although it is possible to have multiple web modules in a single application, the simple scaffolding starts with just one web project.
• the name of the repository project. This contains the third party jars used by the different application modules.
• finally, the name of the test project. The test project is really a convenience from which you can easily run suites of tests for the entire application.

The last time I ran this command, the following output was produced:

ant -f scaffold-build.xml scaffold:create -Dimpala.home=./
Buildfile: scaffold-build.xml

scaffold:input-workspace-root:
[input] Please enter name of workspace root directory:
/Users/philzoio/workspaces/scaffold

scaffold:input-main-project:
[input] Please enter main project name, to be used for root module:
main

scaffold:input-module-project:
[input] Please enter name of first non-root module:
module

scaffold:input-web-project:
[input] Please enter name of web module:
web

scaffold:input-test-project:
[input] Please enter name of tests project:
test

scaffold:input-repository-project:
[input] Please enter name of repository project:
repository

scaffold:create-confirm:
[echo] Workspace root location: /Users/philzoio/workspaces/scaffold
[echo] Main (root) project name: main
[echo] First non-root module project name: module
[echo] Web project name: web
[echo] Tests project name: test
[echo] Repository project name: repository
[input] Press return key to continue, or CTRL + C to quit ...


scaffold:create:
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold
[copy] Copying 6 files to /Users/philzoio/workspaces/scaffold/main
[copy] Copied 6 empty directories to 5 empty directories under /Users/philzoio/workspaces/scaffold/main
[copy] Copying 3 files to /Users/philzoio/workspaces/scaffold/main
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/main/spring
[copy] Copying 4 files to /Users/philzoio/workspaces/scaffold/module
[copy] Copied 5 empty directories to 4 empty directories under /Users/philzoio/workspaces/scaffold/module
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/module
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/module/spring
[copy] Copying 11 files to /Users/philzoio/workspaces/scaffold/web
[copy] Copied 8 empty directories to 4 empty directories under /Users/philzoio/workspaces/scaffold/web
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/web
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/web/spring
[copy] Copying 2 files to /Users/philzoio/workspaces/scaffold/test
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/test
[copy] Copying 1 file to /Users/philzoio/workspaces/scaffold/repository
[copy] Copied 2 empty directories to 1 empty directory under /Users/philzoio/workspaces/scaffold/repository

BUILD SUCCESSFUL

Before we import the Eclipse project, there are just two more steps to follow:

First, go the main newly project, and run the following two commands:

cd /Users/philzoio/workspaces/scaffold/main
ant fetch
ant get

The fetch command will copy the Impala libraries into the repository project of the new workspace, as shown by the following output.

ant fetch
Buildfile: build.xml
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

repository:fetch-impala-from-lib:
[copy] Copying 12 files to /Users/philzoio/workspaces/scaffold/repository/main

repository:fetch-impala-from-repository:

repository:fetch-impala:

fetch:

BUILD SUCCESSFUL

The get command downloads the necessary third party libraries, as defined using a simple format in dependencies.txt files.

ant get
Buildfile: build.xml
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

shared:get:
[echo] Project using workspace.root: /Users/philzoio/workspaces/scaffold
[echo] Project using impala home: /Users/philzoio/impala-SNAPSHOT

download:get:
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold/repository/build
[mkdir] Created dir: /Users/philzoio/workspaces/scaffold/repository/test
[get] Getting: http://ibiblio.org/pub/packages/maven2/commons-logging/commons-logging/1.1/commons-logging-1.1.jar
[get] To: /Users/philzoio/workspaces/scaffold/repository/main/commons-logging-1.1.jar
[get] Getting: http://ibiblio.org/pub/packages/maven2/commons-logging/commons-logging/1.1/commons-logging-1.1-sources.jar
[get] To: /Users/philzoio/workspaces/scaffold/repository/main/commons-logging-1.1-sources.jar

...

[download] ******************************************************
[download]
[download]                RESULTS OF DOWNLOAD OPERATION
[download]
[download] org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2.jar resolved from
http://ibiblio.org/pub/packages/maven2/org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2.jar
[download] org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2-sources.jar resolved from
http://ibiblio.org/pub/packages/maven2/org/springframework/spring-webmvc/2.5.2/spring-webmvc-2.5.2-sources.jar
[download]
[download] ******************************************************

get:

BUILD SUCCESSFUL
Total time: 3 minutes 6 seconds

We're now ready to import our projects into Eclipse. Start by opening Eclipse in the newly created workspace.

Use the menus File -> Import ... -> General -> Existing Projects Into Workspace. When prompted, set the import base directory to the workspace root directory. This should bring up a dialog box as shown below.

Select all of the projects and import them.

If you reach this point and no errors are showing in your workspace, the congratulations! You have just set up a new Impala workspace.

Let's test it out:

Running up the web application

Using CTRL-Shift + T, find the class StartServer. Right click, then select Run As ... Java Application.

This will start a Jetty Server and run up a server on port 8080.

The text shown on the console view of Eclipse will look something like this:

2008-04-01 20:56:10.408::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2008-04-01 20:56:10.479::INFO:  jetty-6.1.1
2008-04-01 20:56:10.963:/web:INFO:  Initializing Spring root WebApplicationContext
INFO : BaseImpalaContextLoader - Loading bootstrap context from locations [META-INF/impala-bootstrap.xml, META-INF/impala-web-bootstrap.xml, META-INF/impala-jmx-bootstrap.xml, META-INF/impala-web-listener-bootstrap.xml]
INFO : ScheduledModuleChangeMonitor - Starting org.impalaframework.module.monitor.ScheduledModuleChangeMonitorBean with fixed delay of 2 and interval of 10
INFO : LoadTransitionProcessor - Loading definition root-module
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module root-module: [file [/Users/philzoio/workspaces/scaffold/main/bin]]
INFO : LoadTransitionProcessor - Loading definition module
INFO : ModuleContributionPostProcessor - Contributing bean messageService from module module
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module module: [file [/Users/philzoio/workspaces/scaffold/module/bin]]
INFO : LoadTransitionProcessor - Loading definition web
INFO : ScheduledModuleChangeMonitor - Monitoring for changes in module web: [file [/Users/philzoio/workspaces/scaffold/web/bin]]
2008-04-01 20:56:12.133:/web:INFO:  Initializing Spring FrameworkServlet 'web'
INFO : ExternalLoadingImpalaServlet - FrameworkServlet 'web': initialization started
INFO : ExternalLoadingImpalaServlet - FrameworkServlet 'web': initialization completed in 21 ms
2008-04-01 20:56:12.168::INFO:  Started SelectChannelConnector @ 0.0.0.0:8080
DEBUG : ScheduledModuleChangeMonitor - Completed check for modified modules. No modified module contents found
DEBUG : ScheduledModuleChangeMonitor - Completed check for modified modules. No modified module contents found

You can connect to the server using the URL:

http://localhost:8080/web/message.htm



Note that Impala is started up to automatically detect changes in your modules, and reload modules in response to these changes. You can play with this mechanism by making changes to classes such as MessageController (in the web project) and MessageServiceImpl (in the module project).

Running the standalone interactive client

Use Eclipse to find the JUnit test class MessageIntegrationTest. Again, run this as a Java application, and execute tests, reload modules etc, using the interactive test runner. Here's some example output:

log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
Test class set to test.MessageIntegrationTest
Unable to load module corresponding with directory name [not set]
Starting inactivity checker with maximum inactivity of 600 seconds
--------------------

Please enter your command text
>test
No module loaded for current directory: main
Running test testIntegration
.Hello World!

Time: 0.055

OK (1 test)


Please enter your command text
>reload
Module 'root-module' loaded in 0.096 seconds
Used memory: 1.5MB
Max available memory: 63.6MB


Please enter your command text
>module module
Module 'module' loaded in 0.035 seconds
Used memory: 2.1MB
Max available memory: 63.6MB


Please enter your command text
>t
No module loaded for current directory: main
Running test testIntegration
.Hello World!

Time: 0.012

OK (1 test)

Run the suite of tests

Any of the JUnit integration tests can be run as a regular unit test in Eclipse, with green bar and all. From the tests project, find the class AllTests. This contains a suite of tests covering all the tests in the project. Run this as a regular unit test, and you will see the following:



There's plenty more to get your teeth stuck into, but having a working web application, a working suite of tests, and interactive tests which can be run out the box is a helpful start.

Impala working on Tomcat as war

Today I reached a significant milestone with Impala. I successfully deployed and ran the Impala as a war file with reloadable modules, in Tomcat. Modules are placed as jars in the folder /WEB-INF/modules.

This is a big step forward. Until now, Impala has been developed and deployed in Eclipse or using ANT and an embedded Jetty runner. That's great for development, but for production, a war file is much more convenient - just having a single file which can be dropped into a web container is much more deployment friendly than having to set up a special folder on the target machine which has a structure which mirrors the development environment.

A war deployment option was one of the main features which stands in the way of a public release. There's quite a bit of tidying up to do in the way that the build/packaging process works, but it's fair to say that we're now a big step closer to a first public release of Impala.

Modularity: what is it, and why is it important?

Modularity is taken for granted as something which has value in software engineering, perhaps without always a well articulated understanding of what it is, and why it is valuable. I came to this realisation when I found myself having in some difficulty attempting to explain why the modularity that Impala brings can be beneficial to a large application. Instinctively, I felt like I had a good understanding, but it's useful to put this understanding into words.

What is modularity?

I googled remarkably few articles dedicated to the subject. The definition in the wikipedia entry was quite good, from which I quote the relevant sections:

"Programs that have many direct interrelationships between any two random parts of the program code are less modular (more tightly coupled) than programs where those relationships occur mainly at well-defined interfaces between modules.
Modules provide a separation between interface and implementation. A module interface expresses the elements that are provided and required by the module. The elements defined in the interface are visible to other modules. The implementation contains the working code that corresponds to the elements declared in the interface."

Modularity in Impala

This definition fits in very well with how modularity is implemented in Impala. Modules fit into a hierarchy. An Impala module only depends directly on its parent. This means that two modules at the same level have no direct dependencies on each other. Modules communicate with each other through well defined interfaces - specifically Java interfaces defined in a shared parent module.

Child modules dynamically contribute their implementation to a proxy bean defined in a higher level. I'm also planning contributions to a service registry, as is the case with OSGi. The proxies can themselves be dynamically registered, although static wirings of these are necessary to support the traditional named bean style dependency injection.

Why is Modularity Important?

Modularity is a key weapon in reducing with complexity in a large project. In the same way that blocks of functionality within a component should be partitioned into classes according to responsibilities, so should parts of an application be partitioned into modules. Like classes in an OO model, each module should have a well defined - albeit higher level - set of responsibilities. Ideally, these responsibilities should be closely related to each other.

The benefits of modularity, in my view, are as follows:

• by definition, it reduces the unnecessary cross dependencies between code for different parts of a system. This makes code easier to maintain and extend. Impala enforces this kind of modularity because classes at the same level of the module hierarchy are not visible to each other.
  
• it's easier to specify the constituents of a modular system at a higher level. An Impala application is just a collection of modules. In practical terms, it is much more convenient to specify an Impala application as a small number of modules than as a large number of Spring configuration files.
• it's easier to read and understand a modular system. Each module has a clear and relatively narrowly defined set of functions, which can be more easily documented and explained than in the case of a system which is not modular.
• a dynamic modular system such as Impala also has the advantage that parts of the system can be separately added, upgraded and removed. This is not possible in a system which is not modular.

Without modularity, a large system rapidly degenerates into one which is extremely hard to maintain. I would go so far as to argue that a large system which is not modular is doomed to excessive complexity, maintenance issues, and an unnecessarily high bug rate.