The JVM Test Suite plugin (plugin id: jvm-test-suite) provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects. Tests suites are intended to grouped by their purpose and can have separate dependencies and use different testing frameworks.

For instance, this plugin can be used to define a group of Integration Tests, which might run much longer than unit tests and have different environmental requirements.

Usage

This plugin is applied automatically by the java plugin but can be additionally applied explicitly if desired. The plugin cannot be used without a JVM language plugin applied as it relies on several conventions of the java plugin.

Example 1. Applying the JVM Test Suite plugin
build.gradle
plugins {
    id 'java'
    id 'jvm-test-suite'
}
build.gradle.kts
plugins {
    java
    `jvm-test-suite`
}

The plugins adds the following objects to the project:

  • A testing extension (type: TestingExtension) to the project used to configure test suites.

When used with the Java Plugin:

  • A test suite named test (type: JvmTestSuite).

  • A test SourceSet.

  • Several configurations derived from the test SourceSet name: testImplementation, testCompileOnly, testRuntimeOnly

  • A single test suite target backed by a task named test.

The test task, SourceSet and derived configurations are identical in name and function to those used in prior Gradle releases.

Tasks

The JVM Test Suite plugin adds the following task to the project:

test — Test

Depends on: testClasses from the java plugin, and all tasks which produce the test runtime classpath

Runs the tests using JUnit by default but can be configured for other frameworks.

Additional instances of Test tasks will be automatically created for each test suite added via the testing extension.

Configuration

See the TestingExtension class in the API documentation.

Terminology and Modeling

The JVM Test Suite Plugin introduces some modeling concepts backed by new APIs. Here are their definitions.

Test Suite

A test suite is a collection of JVM-based tests.

Test Suite Target

For the initial release of this plugin, each test suite has a single target. This results in a 1:1:1 relationship between test suite, test suite target and a matching Test task. The name of the Test task is derived from the suite name. Future iterations of the plugin will allow defining multiple targets based other attributes, such as a particular JDK/JRE runtime.

Each test suite has some configuration that is common across for all tests contained in the suite:

  • Testing framework

  • Sources

  • Dependencies

In the future, other properties may be specified in the test suite which may influence the toolchains selected to compile and run tests.

The Test task associated with the target inherits its name from the suite. Other properties of the Test task are configurable.

Configuration Examples

Here are several examples to illustrate the configurability of test suites.

Declare an additional test suite

build.gradle
testing {
    suites { (1)
        test { (2)
            useJUnitJupiter() (3)
        }

        integrationTest(JvmTestSuite) { (4)
            dependencies {
                implementation project (5)
            }

            targets { (6)
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
        }
    }
}

tasks.named('check') { (7)
    dependsOn(testing.suites.integrationTest)
}
build.gradle.kts
testing {
    suites { (1)
        val test by getting(JvmTestSuite::class) { (2)
            useJUnitJupiter() (3)
        }

        val integrationTest by registering(JvmTestSuite::class) { (4)
            dependencies {
                implementation(project) (5)
            }

            targets { (6)
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
        }
    }
}

tasks.named("check") { (7)
    dependsOn(testing.suites.named("integrationTest"))
}
1 Configure all test suites for this project.
2 Configure the built-in test suite. This suite is automatically created for backwards compatibility. By convention, the built-in test suite will use JUnit4.
3 Declare this test suite uses JUnit Jupiter. The framework’s dependencies are automatically included. It is not necessary to explicitly name or configure the built-in test suite if JUnit4 is desired.
4 Define a new suite called integrationTest.
5 Add a dependency on the production code of the project to the integrationTest suite targets. By convention, only the built-in test suite will automatically have a dependency on the production code of the project.
6 Configure all targets of this suite. By convention, test suite targets have no relationship to other Test tasks. This example shows that the slower integration test suite targets should run after all targets of the test suite are complete.
7 Configure the check task to depend on all integrationTest targets. By convention, test suite targets are not associated with the check task.

Invoking the check task on the above configured build should show output similar to:

> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses
> Task :test
> Task :jar
> Task :compileIntegrationTestJava
> Task :processIntegrationTestResources NO-SOURCE
> Task :integrationTestClasses
> Task :integrationTest
> Task :check

BUILD SUCCESSFUL in 0s
6 actionable tasks: 6 executed

Note that the integrationTest test suite does not run until after the test test suite completes.

Configure the built-in test suite

build.gradle
testing { (1)
    suites {
        test {
            useTestNG() (1)

            targets {
                all {
                    testTask.configure { (2)
                        // set a system property for the test JVM(s)
                        systemProperty 'some.prop', 'value'
                        options { (3)
                            preserveOrder = true
                        }
                    }
                }
            }
        }
    }
}
build.gradle.kts
testing {
    suites {
        val test by getting(JvmTestSuite::class) {
            useTestNG() (1)

            targets {
                all {
                    testTask.configure { (2)
                        // set a system property for the test JVM(s)
                        systemProperty("some.prop", "value")
                        options { (3)
                            val options = this as TestNGOptions
                            options.preserveOrder = true
                        }
                    }
                }
            }
        }
    }
}
1 Declare the test test suite uses the TestNG test framework.
2 Lazily configure the test task of all targets of the suite; note the return type of testTask is TaskProvider<Test>.
3 Configure more detailed test framework options. The options will be a subclass of org.gradle.api.tasks.testing.TestFrameworkOptions, in this case it is org.gradle.api.tasks.testing.testng.TestNGOptions.

Configure dependencies of a test suite

build.gradle
testing {
    suites {
        test { (1)
            dependencies {
                implementation 'org.assertj:assertj-core:3.21.0' (2)
            }
        }
    }
}

// Note that this is equivalent to:
dependencies {
    testImplementation 'org.assertj:assertj-core:3.21.0'
}
build.gradle.kts
testing {
    suites {
        val test by getting(JvmTestSuite::class) { (1)
            dependencies {
                implementation("org.assertj:assertj-core:3.21.0") (2)
            }
        }
    }
}

// Note that this is equivalent to:
dependencies {
    testImplementation("org.assertj:assertj-core:3.21.0")
}
1 Configure the built-in test test suite.
2 Add the assertj library to the test’s compile and runtime classpaths. The dependencies block within a test suite is already scoped for that test suite. Instead of having to know the global name of the configuration, test suites have a consistent name that you use in this block for declaring implementation, compileOnly and runtimeOnly dependencies.

Configure source directories of a test suite

build.gradle
testing {
    suites {
        integrationTest(JvmTestSuite) { (1)
            sources { (2)
                java { (3)
                    srcDirs = ['src/it/java'] (4)
                }
            }
        }
    }
}
build.gradle.kts
testing {
    suites {
        val integrationTest by registering(JvmTestSuite::class) { (1)
            sources { (2)
                java { (3)
                    setSrcDirs(listOf("src/it/java")) (4)
                }
            }
        }
    }
}
1 Declare and configure a suite named integrationTest. The SourceSet and synthesized Test tasks will be based on this name.
2 Configure the sources of the test suite.
3 Configure the java SourceDirectorySet of the test suite.
4 Overwrite the srcDirs property, replacing the conventional src/integrationTest/java location with src/it/java.

Configure the Test task for a test suite

build.gradle
testing {
    suites {
        integrationTest {
            targets {
                all { (1)
                    testTask.configure {
                        maxHeapSize = '512m' (2)
                    }
                }
            }
        }
    }
}
build.gradle.kts
testing {
    suites {
        val integrationTest by getting(JvmTestSuite::class) {
            targets {
                all { (1)
                    testTask.configure {
                        setMaxHeapSize("512m") (2)
                    }
                }
            }
        }
    }
}
1 Configure the integrationTest task created by declaring a suite of the same name.
2 Configure the Test task properties.

Test tasks associated with a test suite target can also be configured directly, by name. It is not necessary to configure via the test suite DSL.

Differences between similar methods on JvmTestSuite and Test task types

Instances of JvmTestSuite have methods useJUnit() and useJUnitJupiter(), which are similar in name to methods on the Test task type: useJUnit() and useJUnitPlatform(). However, there are important differences.

Unlike the methods on the Test task, JvmTestSuite methods perform two additional pieces of configuration:

  1. JvmTestSuite#useJUnit(), #useJUnitJupiter() and other framework-specific methods automatically apply the relevant testing framework libraries to the compile and runtime classpaths of the suite targets. Note the overloaded methods like JvmTestSuite#useJUnit(String) and #useJUnitJupiter(String) allow you to provide specific versions of the framework dependencies.

  2. JvmTestSuite#useJUnit() and #useJUnitJupiter() automatically configure the suite targets' Test tasks to execute using the specified framework.

An example of this behavior is shown in the first configuration example above, Configuring the built-in test suite.

Note that unlike the Test task, the aforementioned methods on JvmTestSuite do not accept a closure or action for configuring the framework. These framework configuration options can be set on the individual targets.