Sometimes we can fall into a false sense of security when we see all our unit tests pass. But sometimes we forget that the application must also run, and unit tests do not cover that.

Urgh.

There was a refactoring push recently to smooth out the build process for a web application I work with. The .war file being generated was getting bloated, and it was due to many unneeded .jar files being included during the build. The refactoring process involved using some dependency-checking functions with maven. It was a labor-intensive and iterative process. After a few days, it was thought that the effort had completed. After all, maven was no longer complaining about either unreferenced .jars, or required .jars not being found. Not only were there no compile errors, there weren’t any warnings, either. Things looked very good.

That is, until we deployed and tried to start the server. The application server barfed almost immediately after starting the application. The problem: overzealous removal of .jar files from the project’s .pom file. The root problem was that we were looking at compile-time dependencies to determine if .jar files could be removed, we weren’t taking runtime dependencies into account. The fix was to include the dependent .jar files with a “runtime” scope in our .pom files.

The moral of this story is that while unit tests are great, and an amazing tool to reduce project defect count (particularly when used in conjunction with Test Driven Development), it’s not the end-all test tool. In this case, the .pom files, as well as any other changes from the refactor, should not have been committed to source control until after the application was actually deployed (locally, on the developer’s workstation), and put through it’s paces.