Testing plays a crucial role in the development process as it ensures reliable and high-quality software. The same principles apply to build code and more specifically Gradle plugins. In this section you will learn effective techniques for testing plugin code.

This section assumes you have:

The sample project

All discussions in this section are centered around a sample project called URL verifier plugin. The plugin creates a task named verifyUrl that checks whether a given URL can be resolved via HTTP GET. The end user can provide the URL via an extension named verification.

The following build script assumes that the plugin JAR file has been published to a binary repository. In a nutshell, the script demonstrates how to apply the plugin to the project and configure its exposed extension.

build.gradle
plugins {
    id 'org.myorg.url-verifier'         (1)
}

verification {
    url = 'https://www.google.com/'     (2)
}
build.gradle.kts
plugins {
    id("org.myorg.url-verifier")        (1)
}

verification {
    url.set("https://www.google.com/")  (2)
}
1 Applies the plugin to the project
2 Configures the URL to be verified through the exposed extension

Executing the task renders a success message if the HTTP GET call to the configured URL returns with a 200 response code.

$ gradle verifyUrl

> Task :verifyUrl
Successfully resolved URL 'https://www.google.com/'

BUILD SUCCESSFUL in 0s
5 actionable tasks: 5 executed

Before diving into the code, let’s first revisit the different types of tests and the tooling that supports implementing them.

On the importance of testing

Testing is a foundational activity in the software development life cycle. Appropriate testing ensures that the software works on a functional and non-functional level before it is released to the end user. As a by product, automated testing also enables the development team to refactor and evolve the code without fearing to introduce regressions in the process.

The testing pyramid

testing pyramid

Probably the easiest way to test software is to manually exercise it. Manual testing can occur at any time and is not bound to writing automation code. However, manual testing is error-prone and cumbersome as it requires a human to walk through a set of predefined test cases. Manually testing Gradle plugins requires consuming the plugin binary in a build script.

Other types of tests can be fully automated and exercised with every change to the source code. The testing pyramid introduced by Mike Cohen in his book Succeeding with Agile: Software Development Using Scrum describes three types of automated tests.

Unit testing aims to verify the smallest unit of code. In Java-based projects this unit is a method. Unit tests usually do not interact with other parts of the system e.g. a database or the file system. Interactions with other parts of the system are usually cut off with the help of Stubs or Mocks. You will find that POJOs and utility classes are good candidates for unit tests as they are self-contained and do not use the Gradle API.

Integration testing verifies that multiple classes or components work together as a whole. The code under test may reach out to external subsystems.

Functional testing is used to test the system from the end user’s perspective. End-to-end tests for Gradle plugins stand up a build script, apply the plugin under test and execute the build with a specific task. The outcome of the build (e.g. standard output/error or generated artifacts) verifies the correctness of the functionality.

Tooling support

Implementing manual and automated testing for Gradle plugins is straight forward - it just requires the right tooling. The table below gives you a brief overview on how to approach each test type. Please be aware that you have the free choice of using the test framework you are most familiar with. For a detailed discussion and code example please refer to the dedicated section further down.

Test type Tooling support

Manual tests

Gradle composite builds

Unit tests

Any JVM-based test framework

Integration tests

Any JVM-based test framework

Functional tests

Any JVM-based test framework and Gradle TestKit

Setting up manual tests

The composite builds feature of Gradle makes it very easy to test a plugin manually. The standalone plugin project and the consuming project can be combined together into a single unit making it much more straight forward to try out or debug changes without the hassle of re-publishing the binary file.

.
├── include-plugin-build   (1)
│   ├── build.gradle
│   └── settings.gradle
└── url-verifier-plugin    (2)
    ├── build.gradle
    ├── settings.gradle
    └── src
1 Consuming project that includes the plugin project
2 The plugin project

There are two ways to include a plugin project into a consuming project.

1. By using the command line option --include-build. 2. By using the method includeBuild in settings.gradle.

The following code snippet demonstrates the use of the settings file.

settings.gradle
pluginManagement {
    includeBuild '../url-verifier-plugin'
}
settings.gradle.kts
pluginManagement {
    includeBuild("../url-verifier-plugin")
}

The command line output of task verifyUrl from the project include-plugin-build looks exactly the same as shown in the introduction except that it now executed as part of a composite build.

Manual testing has its place in the development process. By no means is it a replacement for automated testing. Next up, you’ll learn how to organize and implement automated tests for Gradle plugins.

Setting up automated tests

Setting up a suite of tests earlier on is crucial to the success of your plugin. You will encounter various situations that make your tests an invaluable safety net you can rely on e.g. when upgrading the plugin to a new Gradle version and enhancing or refactoring the code.

Organizing test source code

We recommend to implement a good distribution of unit, integration and functional tests to cover the most important use cases. Separating the source code for each test type automatically results in a project that is more maintainable and manageable.

By default the Java project already creates a convention for organizing unit tests, the directory src/test/java. Additionally, if you apply the Groovy plugin source code under the directory src/test/groovy is taken under consideration for compilation. Consequently, source code directories for other test types should follow a similar pattern. Below you can find an exemplary project layout for a plugin project that chooses to use a Groovy-based testing approach.

.
└── src
    ├── functionalTest
    │   └── groovy      (1)
    ├── integrationTest
    │   └── groovy      (2)
    ├── main
    │   ├── java        (3)
    └── test
        └── groovy      (4)
1 Source directory containing functional tests
2 Source directory containing integration tests
3 Source directory containing production source code
4 Source directory containing unit tests
The directories src/integrationTest/groovy and src/functionalTest/groovy are not based on an existing standard convention for Gradle projects. You are free to choose any project layout that works best for you.

In the next section, you will learn how to configure those source directories for compilation and test execution. You can also rely on third-party plugins for convience e.g. the Nebula Facet plugin or the TestSets plugin.

Modeling test types

A new configuration DSL for modeling the below integrationTest suite is available via the incubating JVM Test Suite plugin.

Gradle models source code directories with the help of the source set concept. By pointing an instance of a source set to one or many source code directories, Gradle will automatically create a corresponding compilation task out-of-the-box. A pre-configured source set can be created with one line of build script code. The source set automatically registers configurations to define dependencies for the sources of the source set. We use that to define an integrationTestImplementation dependency to the project itself, which represents the "main" variant of our project (i.e. the compiled plugin code).

build.gradle
def integrationTest = sourceSets.create("integrationTest")

dependencies {
    integrationTestImplementation(project)
}
build.gradle.kts
val integrationTest by sourceSets.creating

dependencies {
    "integrationTestImplementation"(project)
}

Source sets are only responsible for compiling source code, but do not deal with executing the byte code. For the purpose of test execution, a corresponding task of type Test needs to be established. The following listing shows the setup for executing integration tests. As you can see below, the task references the classes and runtime classpath of the integration test source set.

build.gradle
def integrationTestTask = tasks.register("integrationTest", Test) {
    description = 'Runs the integration tests.'
    group = "verification"
    testClassesDirs = integrationTest.output.classesDirs
    classpath = integrationTest.runtimeClasspath
    mustRunAfter(tasks.named('test'))
}
tasks.named('check') {
    dependsOn(integrationTestTask)
}
build.gradle.kts
val integrationTestTask = tasks.register<Test>("integrationTest") {
    description = "Runs the integration tests."
    group = "verification"
    testClassesDirs = integrationTest.output.classesDirs
    classpath = integrationTest.runtimeClasspath
    mustRunAfter(tasks.test)
}
tasks.check {
    dependsOn(integrationTestTask)
}

Configuring a test framework

Gradle does not dictate the use of a specific test framework. Popular choices include JUnit, TestNG and Spock. Once you choose an option, you have to add its dependency to the compile classpath for your tests. The following code snippet shows how to use Spock for implementing tests. We choose to use it for all our three test types (test, integrationTest and functionalTest) and thus define a dependency for each of them.

build.gradle
repositories {
    mavenCentral()
}

dependencies {
    testImplementation platform("org.spockframework:spock-bom:2.0-groovy-3.0")
    testImplementation 'org.spockframework:spock-core'

    integrationTestImplementation platform("org.spockframework:spock-bom:2.0-groovy-3.0")
    integrationTestImplementation 'org.spockframework:spock-core'

    functionalTestImplementation platform("org.spockframework:spock-bom:2.0-groovy-3.0")
    functionalTestImplementation 'org.spockframework:spock-core'
}

tasks.withType(Test).configureEach {
    // Using JUnitPlatform for running tests
    useJUnitPlatform()
}
build.gradle.kts
repositories {
    mavenCentral()
}

dependencies {
    testImplementation(platform("org.spockframework:spock-bom:2.0-groovy-3.0"))
    testImplementation("org.spockframework:spock-core")

    "integrationTestImplementation"(platform("org.spockframework:spock-bom:2.0-groovy-3.0"))
    "integrationTestImplementation"("org.spockframework:spock-core")

    "functionalTestImplementation"(platform("org.spockframework:spock-bom:2.0-groovy-3.0"))
    "functionalTestImplementation"("org.spockframework:spock-core")
}

tasks.withType<Test>().configureEach {
    // Using JUnitPlatform for running tests
    useJUnitPlatform()
}
Spock is a Groovy-based BDD test framework that even includes APIs for creating Stubs and Mocks. The Gradle team prefers Spock over other options for its expressiveness and conciseness.

Implementing automated tests

This section discusses representative implementation examples for unit, integration and functional tests. All test classes are based on the use of Spock though it should be relatively easy to adapt the code to a different test framework. Please revisit the section the testing pyramid for a formal discussion of the definition of each test type.

Implementing unit tests

The URL verifier plugin emits HTTP GET calls to check if a URL can be resolved successfully. The method DefaultHttpCaller.get(String) is responsible for calling a given URL and returns with an instance of type HttpResponse. HttpResponse is a POJO containing information about the HTTP response code and message.

HttpResponse.java
package org.myorg.http;

public class HttpResponse {
    private int code;
    private String message;

    public HttpResponse(int code, String message) {
        this.code = code;
        this.message = message;
    }

    public int getCode() {
        return code;
    }

    public String getMessage() {
        return message;
    }

    @Override
    public String toString() {
        return "HTTP " + code + ", Reason: " + message;
    }
}

The class HttpResponse represents a good candidate to be tested by a unit test. It does not reach out to any other classes nor does it use the Gradle API.

HttpResponseTest.groovy
package org.myorg.http

import spock.lang.Specification

class HttpResponseTest extends Specification {

    private static final int OK_HTTP_CODE = 200
    private static final String OK_HTTP_MESSAGE = 'OK'

    def "can access information"() {
        when:
        def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)

        then:
        httpResponse.code == OK_HTTP_CODE
        httpResponse.message == OK_HTTP_MESSAGE
    }

    def "can get String representation"() {
        when:
        def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)

        then:
        httpResponse.toString() == "HTTP $OK_HTTP_CODE, Reason: $OK_HTTP_MESSAGE"
    }
}
When writing unit tests, it’s important to test boundary conditions and various forms of invalid input. Furthermore, try to extract as much logic as possible from classes that use the Gradle API to make it testable as unit tests. It will buy you the benefit of maintainable code and faster test execution.

Implementing integration tests

Let’s have a look at a class that reaches out to another system, the piece of code that emits the HTTP calls. At the time of executing a test for the class DefaultHttpCaller, the runtime environment needs to be able to reach out to the internet.

DefaultHttpCaller.java
package org.myorg.http;

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;

public class DefaultHttpCaller implements HttpCaller {
    @Override
    public HttpResponse get(String url) {
        try {
            HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
            connection.setConnectTimeout(5000);
            connection.setRequestMethod("GET");
            connection.connect();

            int code = connection.getResponseCode();
            String message = connection.getResponseMessage();
            return new HttpResponse(code, message);
        } catch (IOException e) {
            throw new HttpCallException(String.format("Failed to call URL '%s' via HTTP GET", url), e);
        }
    }
}

Implementing an integration test for DefaultHttpCaller doesn’t look much different from the unit test shown in the previous section.

DefaultHttpCallerIntegrationTest.groovy
package org.myorg.http

import spock.lang.Specification
import spock.lang.Subject

class DefaultHttpCallerIntegrationTest extends Specification {
    @Subject HttpCaller httpCaller = new DefaultHttpCaller()

    def "can make successful HTTP GET call"() {
        when:
        def httpResponse = httpCaller.get('https://www.google.com/')

        then:
        httpResponse.code == 200
        httpResponse.message == 'OK'
    }

    def "throws exception when calling unknown host via HTTP GET"() {
        when:
        httpCaller.get('https://www.wedonotknowyou123.com/')

        then:
        def t = thrown(HttpCallException)
        t.message == "Failed to call URL 'https://www.wedonotknowyou123.com/' via HTTP GET"
        t.cause instanceof UnknownHostException
    }
}

Implementing functional tests

Functional tests verify the correctness of the plugin end-to-end. In practice that means applying, configuring and executing the functionality of the plugin implementation represented by the class UrlVerifierPlugin. As you can see, the implementation exposes an extension and a task instance that uses the URL value configured by the end user.

UrlVerifierPlugin.java
package org.myorg;

import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.myorg.tasks.UrlVerify;

public class UrlVerifierPlugin implements Plugin<Project> {
    @Override
    public void apply(Project project) {
        UrlVerifierExtension extension = project.getExtensions().create("verification", UrlVerifierExtension.class);
        UrlVerify verifyUrlTask = project.getTasks().create("verifyUrl", UrlVerify.class);
        verifyUrlTask.getUrl().set(extension.getUrl());
    }
}

Every Gradle plugin project should apply the plugin development plugin to reduce boilerplate code. By applying the plugin development plugin, the test source set is preconfigured for the use with TestKit. If we want to use a custom source set for functional tests and leave the default test source set for only unit tests, we can configure the plugin development plugin to look for TestKit tests elsewhere.

build.gradle
gradlePlugin {
    testSourceSets(sourceSets.functionalTest)
}
build.gradle.kts
gradlePlugin {
    testSourceSets(functionalTest)
}

Functional tests for Gradle plugins use an instance of GradleRunner to execute the build under test. GradleRunner is an API provided by TestKit which internally uses the Tooling API to execute the build. The following example applies the plugin to the build script under test, configures the extension and executes the build with the task verifyUrl. Please see the TestKit documentation to get more familiar with the functionality of TestKit.

UrlVerifierPluginFunctionalTest.groovy
package org.myorg

import org.gradle.testkit.runner.GradleRunner
import spock.lang.Specification
import spock.lang.TempDir

import static org.gradle.testkit.runner.TaskOutcome.SUCCESS

class UrlVerifierPluginFunctionalTest extends Specification {
    @TempDir File testProjectDir
    File buildFile

    def setup() {
        buildFile = new File(testProjectDir, 'build.gradle')
        buildFile << """
            plugins {
                id 'org.myorg.url-verifier'
            }
        """
    }

    def "can successfully configure URL through extension and verify it"() {
        buildFile << """
            verification {
                url = 'https://www.google.com/'
            }
        """

        when:
        def result = GradleRunner.create()
            .withProjectDir(testProjectDir)
            .withArguments('verifyUrl')
            .withPluginClasspath()
            .build()

        then:
        result.output.contains("Successfully resolved URL 'https://www.google.com/'")
        result.task(":verifyUrl").outcome == SUCCESS
    }
}

IDE integration

TestKit determines the plugin classpath by running a specific Gradle task. You will need to execute the assemble task to initially generate the plugin classpath or to reflect changes to it even when running TestKit-based functional tests from the IDE.

Some IDEs provide a convenience option to delegate the "test classpath generation and execution" to the build. In IntelliJ you can find this option under Preferences…​ > Build, Execution, Deployment > Build Tools > Gradle > Runner > Delegate IDE build/run actions to gradle.

intellij delegate to build