Why is memory never enough?
Our customer project has a few dozens of integration JUnit tests. Those are tests coded as JUnit tests loading the core application Spring context to verify the correct behavior of certain parts of our application. They have been pretty useful proving us right and a lot of times wrong after we have committed something into our source code repository.
The only problem that we were facing was that the tests were really slow. They would take 6, 8, even 12 hours sometimes in a Hudson build. That was a little suspicious and by taking a closer look at one of the integration test runs using Visual VM we managed to find out that we were having a memory leak. The max heap size would reach 1.5GB and because of the huge overload of garbage collection our tests were taking a lot more time than they should have. The JVM was trying hard to collect whatever it could to fit the tests into this pretty big heap.
One interesting thing was that after moving the project from JDK 5 to JDK 6 the tests started failing much quicker with an OutOfMemoryError and they were taking 2 hours less in total. The error would print the mysterious
java.lang.OutOfMemoryError: GC overhead limit exceeded.
It was the first time I had seen such an error and after some googling it turned out that JDK 6 has this new feature to fail the JVM with an out of memory error sooner rather than later.
They also added an option to suppress/enable that feature by adding -XX:-UseGCOverheadLimit to the JVM arguments.
After adding the option -XX:+HeapDumpOnOutOfMemoryError the JVM produced valuable heap dumps on an out of memory errors. Using the Memory Analyzer Tool (MAT) we were able to see where the memory went.
Surprisingly, the JVM was always crashing on two particular test suites. Both of them having 40+ test cases. The biggest heap consumer was an array of objects pointing to instances of either of the test suite classes.
Here's what was uncovered after some investigation:
- JUnit creates a new instance of the test suite class every time a new test case is launched
- JUnit will not invoke TestCase.tearDown() if an exception occurs in TestCase.setUp()
- JUnit keeps reference to all test cases in a test suite until the test suite completes
- Having fields in your test suite class can hold reference to objects that otherwise would be due for garbage collection
tearDown() or rather tearDownHappyCase()
Because our tests had a few fields holding reference to various services and the test application context (based on Spring application context) this was making JUnit overload the memory with ~40MB per test case. Until a test suite would complete we were ending up with 40 test cases x 40MB = ~1.6GB of RAM.
The most obvious way of fixing that was to make sure all the fields in the test class were nullified so that the objects referenced would be garbage collected.
In JUnit tearDown() is the best location for that. The problem was that in case while setting up a test case an error occurs JUnit will continue with the next test case without even bothering invoke tearDown(). This was causing even more troubles with the test cases memory footprint.
Our setUp() method was modified to look as follows:
public void setUp() throws Exception {
try {
super.setUp();
} catch (Throwable exc) {
tearDown();
throw exc;
}
}
And the tearDown() method looked like:
public void tearDown() {
this.productService = null;
this.productSkuService = null;
...
}
Even though not so beautiful this allowed us to get the overall memory usage to reasonable levels and sped up our test cases quite a bit.
JUnit 4
Interestingly enough the test workflow in JUnit 4 is different. There are the @Before and @After annotations to be used on methods in your test case. Even if the method(s) annotated with @Before throw any error JUnit 4 will still execute all the @After annotated methods.
This means the the above mentioned problem will not occur in JUnit 4 test cases.
It's good to know things have been improved...
