Maven Plugin Testing - In a Modern way - Part III

In the second part of the series - Maven Plugin Testing - In a Modern way - Part II we have seen how to make the basic integration test while checking the log output of Maven builds.

In this third part we will dive into how Maven will be called by default during the integration tests and how we can influence that behaviour.

Let us begin with the following basic integration test (we ignore the project which is used for testing at the moment.)

1@MavenJupiterExtension
2class BaseIT {
3
4  @MavenTest
5  void the_first_test_case(MavenExecutionResult result) {
6     ...
7  }
8}

The above test will execute Apache Maven by using the following options by default:

  • --batch-mode
  • --show-version
  • --errors

That means that each execution within the Integration Testing Framework will be done like this:

1mvn --batch-mode --show-version --errors

To get that correctly working and in particular having a local cache for each integration test case the integration testing framework will add: -Dmaven.repo.local=... to each call as well. This is necessary to get the following result:

 1.
 2└──target/
 3   └── maven-it/
 4       └── org/
 5           └── it/
 6               └── FirstMavenIT/
 7                   └── the_first_test_case/
 8                       β”œβ”€β”€ .m2/
 9                       β”œβ”€β”€ project/
10                       β”‚   β”œβ”€β”€ src/
11                       β”‚   β”œβ”€β”€ target/
12                       β”‚   └── pom.xml
13                       β”œβ”€β”€ mvn-stdout.log
14                       β”œβ”€β”€ mvn-stderr.log
15                       └── mvn-arguments.log

The option -Dmaven.repo.local=... can't be changed at the moment. The used command line arguments are wrote into the mvn-arguments.log file which can be consulted for later analysis. So in the end a command line for an integration test looks like this:

1mvn -Dmaven.repo.local=<Directory> --batch-mode --show-version --errors package

The goal which is used (package) and how it can be changed will be handled in Part IV of the series.

So far so good, but sometimes or maybe more often it is needed to add other command line options to a Maven call in particular in relationship with integration tests.

Sometimes your own plugin/extension will printout some useful information on debug level but how to test that? We have mentioned before that the options which are used for an integration test does not contain the debug option.

We can now express the need via this (basic_configuration_with_debug):

 1@MavenJupiterExtension
 2class FailureIT {
 3  ...
 4  @MavenTest
 5  @MavenOption(MavenCLIOptions.DEBUG)
 6  void basic_configuration_with_debug(MavenExecutionResult result) {
 7    assertThat(result)
 8        .isSuccessful()
 9        .out()
10        .info()
11        .containsSubsequence(
12            "--- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven) @ basic_configuration_with_debug ---",
13            "--- jacoco-maven-plugin:0.8.5:prepare-agent (default) @ basic_configuration_with_debug ---",
14            "--- maven-resources-plugin:3.1.0:resources (default-resources) @ basic_configuration_with_debug ---",
15            "--- maven-compiler-plugin:3.8.1:compile (default-compile) @ basic_configuration_with_debug ---",
16            "--- maven-resources-plugin:3.1.0:testResources (default-testResources) @ basic_configuration_with_debug ---",
17            "--- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ basic_configuration_with_debug ---",
18            "--- maven-surefire-plugin:3.0.0-M4:test (default-test) @ basic_configuration_with_debug ---",
19            "--- maven-jar-plugin:3.2.0:jar (default-jar) @ basic_configuration_with_debug ---",
20            "--- maven-site-plugin:3.9.1:attach-descriptor (attach-descriptor) @ basic_configuration_with_debug ---"
21        );
22    assertThat(result)
23        .isSuccessful()
24        .out()
25        .warn()
26        .containsSubsequence(
27            "Neither executionException nor failureException has been set.",
28            "JAR will be empty - no content was marked for inclusion!");
29
30    assertThat(result)
31        .isSuccessful()
32        .out()
33        .debug()
34        .containsSubsequence(
35            "Created new class realm maven.api",
36            "Project: com.soebes.itf.maven.plugin.its:basic_configuration_with_debug:jar:1.0",
37            "Goal:          org.apache.maven.plugins:maven-resources-plugin:3.1.0:resources (default-resources)"
38        );
39
40  }
41}

It's important to say that by using the @MavenOption(..) automatically all other previously mentioned command line options will not being used anymore. In this example the final command line looks like this for the test case basic_configuration_with_debug:

1mvn -Dmaven.repo.local=<path> --debug package

So based on turning on debugging output it means that you can check debugging output like this:

1assertThat(result)
2    .isSuccessful()
3    .out()
4    .debug()
5    .containsSubsequence(
6        "Created new class realm maven.api",
7        "Project: com.soebes.itf.maven.plugin.its:basic_configuration_with_debug:jar:1.0",
8        "Goal:          org.apache.maven.plugins:maven-resources-plugin:3.1.0:resources (default-resources)"
9    );

If you like to have the --batch-mode option, --show-version as well as the --error option back in your test case have to add them like this:

1  @MavenTest
2  @MavenOption(MavenCLIOptions.BATCH_MDOE)
3  @MavenOption(MavenCLIOptions.SHOW_VERSION)
4  @MavenOption(MavenCLIOptions.ERRORS)
5  @MavenOption(MavenCLIOptions.DEBUG)
6  void basic_configuration_with_debug(MavenExecutionResult result) {
7  ...
8  }

The result will be that your Maven command line now looks like before including the supplemental --debug option:

1mvn -Dmaven.repo.local=<Directory> --batch-mode --show-version --errors --debug package

That shows that you can easily combine several command line options for a test case via the @MavenOption annotation.

Some command line options in Maven need supplemental information like --log-file <arg> which requires the name of the log file to redirect all the output into. How can we express this with the @MavenOption annotation? This can simply being achieved like the following:

1  @MavenTest
2  @MavenOption(QUIET)
3  @MavenOption(SHOW_VERSION)
4  @MavenOption(value = LOG_FILE, parameter = "test.log")
5  void basic_configuration_with_debug(MavenExecutionResult result) {
6  ...
7  }

As you can see in the above example you have to give the option via the value of the annotation. The parameter of the option has to be given via parameter. In this case we have used static imports to make it more readable. You can of course work without static imports like this (Just a matter of taste):

1  @MavenTest
2  @MavenOption(MavenCLIOptions.QUIET)
3  @MavenOption(MavenCLIOptions.SHOW_VERSION)
4  @MavenOption(value = MavenCLIOptions.LOG_FILE, parameter = "test.log")
5  void basic_configuration_with_debug(MavenExecutionResult result) {
6  ...
7  }

So what about using the same command line options for several test cases? You can simply add the command line options onto the test class level which looks like this:

 1@MavenJupiterExtension
 2@MavenOption(MavenCLIOptions.FAIL_AT_END)
 3@MavenOption(MavenCLIOptions.NON_RECURSIVE)
 4@MavenOption(MavenCLIOptions.ERRORS)
 5@MavenOption(MavenCLIOptions.DEBUG)
 6class FailureIT {
 7
 8  @MavenTest
 9  void case_one(MavenExecutionResult project) {
10    ..
11  }
12
13  @MavenTest
14  void case_two(MavenExecutionResult result) {
15    ..
16  }
17
18  @MavenTest
19  void case_three(MavenExecutionResult result) {
20    ..
21  }
22
23  @MavenTest
24  void case_four(MavenExecutionResult result) {
25    ..
26  }
27
28}

This means that for each given test case ( case_one, case_two, case_three and case_four) the same set of command line options will be used. Apart from that it is much more convenient to define the command line options only once and not for every single test case.

Wait a second. I want to execute case_four with different command line options? Ok no problem just define the set command line options onto that particular test case like in the following example:

 1@MavenJupiterExtension
 2@MavenOption(MavenCLIOptions.FAIL_AT_END)
 3@MavenOption(MavenCLIOptions.NON_RECURSIVE)
 4@MavenOption(MavenCLIOptions.ERRORS)
 5@MavenOption(MavenCLIOptions.DEBUG)
 6class FailureIT {
 7
 8  @MavenTest
 9  void case_one(MavenExecutionResult project) {
10    ..
11  }
12
13  @MavenTest
14  void case_two(MavenExecutionResult result) {
15    ..
16  }
17
18  @MavenTest
19  void case_three(MavenExecutionResult result) {
20    ..
21  }
22
23  @MavenTest
24  @MavenOption(MavenCLIOptions.DEBUG)
25  void case_four(MavenExecutionResult result) {
26    ..
27  }
28
29}

The test case case_four will not inherit the command line options defined on the class level. It will use only those defined on the test case itself.

Now a usual developer question: I'm really lazy I don't want to write all those four MavenOption annotation for each test class. In particular if I need to change those command line options I have to go through each test class one by one?

There is of course a more convenient solution for that problem. This is called a meta annotation. We can simply create a meta annotation which we call MavenTestOptions. This meta annotation looks like this:

1@Target({ElementType.METHOD, ElementType.TYPE})
2@Retention(RUNTIME)
3@Inherited
4@MavenOption(MavenCLIOptions.FAIL_AT_END)
5@MavenOption(MavenCLIOptions.NON_RECURSIVE)
6@MavenOption(MavenCLIOptions.ERRORS)
7@MavenOption(MavenCLIOptions.DEBUG)
8public @interface MavenTestOptions {
9}

This meta annotation can now being used to change the test class of the previous example like this:

 1@MavenJupiterExtension
 2@MavenTestOptions
 3class FailureIT {
 4
 5  @MavenTest
 6  void case_one(MavenExecutionResult project) {
 7    ..
 8  }
 9
10  @MavenTest
11  void case_two(MavenExecutionResult result) {
12    ..
13  }
14
15  @MavenTest
16  void case_three(MavenExecutionResult result) {
17    ..
18  }
19
20  @MavenTest
21  @MavenOption(MavenCLIOptions.DEBUG)
22  void case_four(MavenExecutionResult result) {
23    ..
24  }
25
26}

The above can be improved a bit more. We can integrate the annotation @MavenJupiterExtension into our self defined meta annotation like this (In the example project I have named the meta annotation @MavenITExecution to have different examples in one project.):

 1@Target({ElementType.METHOD, ElementType.TYPE})
 2@Retention(RUNTIME)
 3@Inherited
 4@MavenJupiterExtension
 5@MavenOption(MavenCLIOptions.FAIL_AT_END)
 6@MavenOption(MavenCLIOptions.NON_RECURSIVE)
 7@MavenOption(MavenCLIOptions.ERRORS)
 8@MavenOption(MavenCLIOptions.DEBUG)
 9public @interface MavenTestOptions {
10}

Based on that we can change the test case to look like this:

 1@MavenTestOptions
 2class FailureIT {
 3
 4  @MavenTest
 5  void case_one(MavenExecutionResult project) {
 6    ..
 7  }
 8
 9  @MavenTest
10  void case_two(MavenExecutionResult result) {
11    ..
12  }
13
14  @MavenTest
15  void case_three(MavenExecutionResult result) {
16    ..
17  }
18
19  @MavenTest
20  @MavenOption(MavenCLIOptions.DEBUG)
21  void case_four(MavenExecutionResult result) {
22    ..
23  }
24
25}

So this it is for Part III. If you like to learn more about the Integration Testing Framework you can consult the users guide. If you like to know the state of the release you can take a look into the release notes.

If you have ideas, suggestions or found bugs please file in an issue on github.

An example of the shown uses cases can be found on GitHub.