Overview

In general having a number of unit- and integration tests in your project (BTW: I hope you have a larger number of tests) is a good thing. Afterwards the question arises: Are those tests enough? Have I covered all possible cases? Now the idea of code coverage comes into my mind. That is a way to check which paths, branches and methods (more accurately: function coverage, statement coverage, branch coverage) etc. have been executed during the tests. That means in other words the higher the rate of coverage is, the more unlikely it is having issues. Or more accurately having issues during the change of code. Yes, some people yell here and say a test does not proof bug free software. Absolutely correct. On the other hand it helps to keep the functionality in your code as it is, even if you change your code (internal implementation, adding new features etc.). So you are not breaking existing functionality (breaking things like that is often called regression).

Based on that, the final question arises: What should be the rate of your coverage? 30%, 50%, 80% or even 100%. Based on experience, I would say 30% is too low and 100% is a bit strange (in such a case you might even test setter's and getter's which does not really makes sense; this means testing the JDK) even if you could reach that level, you seem to be focusing on making the code coverage tool happy. The important part here is to deeply think about your tests and whether what's in your application, is the real critical part. So my practical advice would be ca. 80%. (Yes others might have a different opinion).

The tool to measure code coverage in Java is JaCoCo, but that is not the only one. There are existing things like OpenClover or in Kotlin things like Kover and so on (just to mention some).

I will use here JaCoCo, where also the JaCoCo-Maven-Plugin exists for the usage in your Maven builds. This article will show how to configure the code coverage to finally get the results for unit- and integration-tests.

Example Project

Let's start with an example project which has the general structure like this:

 1src
 2|-- main
 3|   `-- java
 4|       `-- com
 5|           `-- soebes
 6|               `-- example
 7|                   `-- jacoco
 8|                       `-- First.java
 9`-- test
10    `-- java
11        `-- com
12            `-- soebes
13                `-- example
14                    `-- jacoco
15                        |-- FirstIT.java
16                        `-- FirstTest.java

You might already have identified the unit test class based on the naming convention. I strongly recommend to follow that naming convention in any Maven projects. In this example there is a single one FirstTest.java. Furthermore, we have an integration test FirstIT.java. This is a usual directory structure of a Maven project. Also, not to forget, we have to have a pom.xml file, which contains the configuration for the project.

Plugin Definition

We should take a look into the configuration for the project and see, which parts are necessary to configure JaCoCo code coverage within the project. In general, you should define all used plugins within your Maven build to keep your build reproducible. The first thing is to define the jacoco-maven-plugin including its version and groupId. The ellipses ... represent other plugins, which should be defined, but currently are omitted for brevity. Furthermore, the maven-surefire-plugin and maven-failsafe-plugin are required, because the unit tests are executed by the maven-surefire-plugin while the integration tests are executed by the maven-failsafe-plugin.

 1<project...>
 2  ...
 3  <build>
 4    <pluginManagement>
 5      <plugins>
 6        <plugin>
 7          <groupId>org.jacoco</groupId>
 8          <artifactId>jacoco-maven-plugin</artifactId>
 9          <version>0.8.11</version>
10        </plugin>
11        ...
12        <plugin>
13          <groupId>org.apache.maven.plugins</groupId>
14          <artifactId>maven-surefire-plugin</artifactId>
15          <version>3.2.2</version>
16          <configuration>
17            <skipTests>${skipUTs}</skipTests>
18          </configuration>
19        </plugin>
20        <plugin>
21          <groupId>org.apache.maven.plugins</groupId>
22          <artifactId>maven-failsafe-plugin</artifactId>
23          <version>3.2.2</version>
24          <configuration>
25            <skipTests>${skipITs}</skipTests>
26          </configuration>
27        </plugin>
28      </plugins>
29    </pluginManagement>
30    ...
31  </build>
32  ...
33</project...>

So basically we define the versions via pluginManagement and the global configuration. The next thing you need to configure is the jacoco-maven-plugin. In general jacoco works by using agents, which must be executed during the build. That can be accomplished by using the prepare-agent goal. This goal handles that the agent is added to the execution of the maven-surefire-plugin.

 1<project...>
 2  ...
 3  <build>
 4    ...
 5    <plugins>
 6      ..
 7      <plugin>
 8        <groupId>org.jacoco</groupId>
 9        <artifactId>jacoco-maven-plugin</artifactId>
10        <executions>
11          <execution>
12            <goals>
13              <goal>prepare-agent</goal>
14              ...
15            </goals>
16          </execution>
17        </executions>
18      </plugin>
19    </plugins>
20  </build>
21  ...
22</project...>

Examine Execution

If you have applied the previous configuration, the execution can be examined via mvn -X verify and you will find something like this in the output.

1'-javaagent:/.../org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/../target/jacoco.exec'

Be a little careful, because the output of using the -X debugging option is being very long. So best is to redirect the output into a file instead of the console.

The redirection can be done on unix like operating systems (Linux, MacOS) via:

1mvn -X verify | tee mvn.log

and on Windows based machines like this:

1mvn -X verify > mvn.log

Afterward, you can make a deep dive into the resulting output file mvn.log:

This above snippet also shows the resulting jacoco.exec file. This file contains the code coverage (binary format) information.

HTML Report

If you like to produce a human-readable report, you have to add the supplemental report goal which looks like this:

 1<project...>
 2  ...
 3  <build>
 4    ...
 5    <plugins>
 6      ..
 7      <plugin>
 8        <groupId>org.jacoco</groupId>
 9        <artifactId>jacoco-maven-plugin</artifactId>
10        <executions>
11          <execution>
12            <goals>
13              <goal>prepare-agent</goal>
14              <goal>report</goal>
15              ...
16            </goals>
17          </execution>
18        </executions>
19      </plugin>
20    </plugins>
21  </build>
22  ...
23</project...>

The given report goal creates an HTML based report(It includes XML- and CSV-based as well). It is necessary to run mvn verify to create that report, because the report goal is bound to the verify lifecycle-phase by default. I strongly recommend to leave the bindings to the appropriate lifecycle-phase up to the plugins (or more accurate to the plugin authors). In 99.999% of the cases those configurations are correct. There might be exceptions. In such cases you might need to bind the goals to a different lifecycle-phase on your own.

After running mvn verify, the HTML report (including XML and CSV) can be found in the target/site/jacoco/ directory. The HTML report can be watched by opening the index.html file, in a bowser of your choice.

If you don't like to run your whole build up to verify lifecycle-phase, but nevertheless you want to create the JaCoCo reports anyway, this can simply be accomplished by using:

1mvn clean test jacoco:report

That will run the lifecycle up to test phase and separately calling the goal report of the jacoco-maven-plugin (In most cases you can simply omit the clean).

Test Project

At the beginning I wrote about the test project structure, now we focus on the content of the test project. The production code contains a class First.java which contains the following code (JDK17+):

 1public record First(int sum) {
 2
 3  public First add(First first) {
 4    return new First(first.sum + this.sum);
 5  }
 6
 7  public First sub(First first) {
 8    return new First(this.sum - first.sum);
 9  }
10  public First minus(First first) {
11    return new First(this.sum - first.sum);
12  }
13}

Do not judge this code, it is only for demonstration purposes of the code coverage. The unit test class FirstTest.java contains the following:

 1class FirstTest {
 2
 3  @Test
 4  void first_add() {
 5    First sum1 = new First(5);
 6    First sum2 = new First(2);
 7
 8    assertThat(sum1.add(sum2)).isEqualTo(new First(7));
 9  }
10}

and the integration test class FirstIT.java is given as this:

 1class FirstIT {
 2
 3  @Test
 4  void first_minus() {
 5    First sum1 = new First(5);
 6    First sum2 = new First(2);
 7
 8    assertThat(sum1.minus(sum2)).isEqualTo(new First(3));
 9  }
10}

If we have taken a deeper look into the code, you have realized, that the unit test only covers the add(..) method while the integration test covers minus(..) method.

Code Coverage

So if we run mvn clean verify, which means to execute the unit tests. This includes executing of the report goal of JaCoCo Maven Plugin. Let us take a look into the code coverage report:

What you can see is that only the add(..) method has been covered by the tests(Of course the constructor as well). Hold on a second? What about the integration test which should cover the minus(..) method? Interestingly it's not shown as being used which means, there is no test existing. That is a bit scary, because we wrote one FirstIT.java, or we did something else wrong. The simple issue here is that we haven't configured Maven, or more accurately, maven-failsafe-plugin to be executed during the integration test lifecycle phase. So there couldn't be any coverage based on the integration test at all. That issue can be solved by adding supplemental lifecycle binding for the maven-failsafe-plugin like this:

 1<project...>
 2  ...
 3  <build>
 4    ...
 5    <plugins>
 6      ..
 7      <plugin>
 8        <groupId>org.apache.maven.plugins</groupId>
 9        <artifactId>maven-failsafe-plugin</artifactId>
10        <executions>
11          <execution>
12            <goals>
13              <goal>integration-test</goal>
14              <goal>verify</goal>
15            </goals>
16          </execution>
17        </executions>
18      </plugin>
19    </plugins>
20  </build>
21  ...
22</project...>

After adding this, you can execute unit- and integration tests, in one go via mvn clean verify.

As a result of the supplemental configuration, you can see that now, both methods add(...) as well as minus(..) are covered by tests. This is now as expected because add(...) is covered by the unit test FirstTest.java while minus(..) is covered by the integration test FirstIT.java. In the end we have accomplished the mission to get code coverage for our unit- and integration tests within a single build. It's important to mentioned that I don't have added supplemental configuration to reach that point.

Only Unit- or Integration Tests

Based on the given configuration of the maven-surefire-plugin and the maven-failsafe-plugin, you can execute all things separately. So let us execute the unit tests only, but nevertheless creating the reports. To skip the integration tests we can use the defined property skipITs of the configuration which looks like this:

1mvn clean verify -DskipITs

The two defined properties skipITs and skipUTs give us the opportunity to control the execution of either unit- and or integration tests separately. The report goal of jacoco-maven-plugin will be executed at the verify lifecycle phase. If we want to run only the integration tests and report them separately, this can be achieved like this:

1mvn clean verify -DskipUTs

Based on the given setup you can now select what you like to execute unit tests or integration tests or both together. The final result is having also appropriate coverage report.

Conclusion

Unfortunately I often see configurations, where the agent is being configured separately for the plugins (maven-surefire-plugin, maven-failsafe-plugin) while this is not necessary. This is based on very old version of jacoco-maven-plugin less than 0.5.3 (if I correctly remember seven or eight years ago). That means today you don't need a supplemental configuration to handle that correctly.

There are some exceptions, where you have to add a definition for argLine while using maven-surefire-plugin and/or the maven-failsafe-plugin. That's the case while you have to add supplemental configuration to the execution during the tests. That might be while you are using preview language features for JDK21+ or using JDK21 and using things like Mockito (more accurately byte-buddy which used by mockito) where configuration needed to be tweaked like this (Hint: That might change in the future):

 1...
 2      <plugin>
 3        <groupId>org.apache.maven.plugins</groupId>
 4        <artifactId>maven-surefire-plugin</artifactId>
 5        <configuration>
 6          <argLine>
 7            -XX:+EnableDynamicAgentLoading
 8            ...
 9          </argLine>
10        </configuration>
11      </plugin>
12...

The given option -XX:+EnableDynamicAgentLoading is required to suppress WARNINGs like this:

1WARNING: A Java agent has been loaded dynamically (/Users/khm/../net/bytebuddy/byte-buddy-agent/1.14.6/byte-buddy-agent-1.14.6.jar)
2WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
3WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
4WARNING: Dynamic loading of agents will be disallowed by default in a future release

So starting with JDK 21 it is required having such configuration while using Mockito. In consequence, you need a kind of tweaked setup to get a code coverage:

For maven-surefire-plugin and maven-failsafe-plugin there is an option for lazy property evaluation, which means, that properties needed adding to the execution of surefire/failsafe plugin. An example of that is the supplemental configuration to prevent the above warnings.

 1...
 2      <plugin>
 3        <groupId>org.apache.maven.plugins</groupId>
 4        <artifactId>maven-surefire-plugin</artifactId>
 5        <configuration>
 6          <argLine>
 7            @{argLine}
 8            -XX:+EnableDynamicAgentLoading
 9            ...
10          </argLine>
11        </configuration>
12      </plugin>
13...

I will emphasize the @{argLine} in the above configuration. I would suggest to put that configuration at the beginning of the configuration parts.

So if you like to look into a real project setup I can recommend the example project which I have created on Github