Help get this topic noticed by sharing it on Twitter, Facebook, or email.

Add support for integration tests

Integration tests, functional tests, acceptance tests are all common features of a build. It would be nice to capture this in an 'integration-test' plugin and start establishing some conventions around this.

## Implementation ideas

Such a plugin might do something like this:

* Add an `intTest` source set for the test source to live.
* Wire up the compile and runtime dependency configurations for this source set.
* Add an 'intTest` task and wire up its inputs and outputs.
20 people like
this idea
+1
Reply
  • 1
    You mention integration, functional, and acceptance tests, but then describe a plugin that only seems to address one. Would this plugin merely add one more set of tests for integration tests? It seems like it would be nice to have a plugin that could add additional tests of whatever name makes sense. An example of a tests DSL below.


    tests {
    intTest extendsFrom test
    accTest extendsFrom test
    otherTest extendsFrom intTest, accTest
    }
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • The goal would be to end up with something where you could easily stamp out different types of test suites. Something like your example, above.

    These ideas tagged 'contribute' are intended to be (relatively) simple to implement for new contributors to Gradle, where someone could jump in and quickly put together something useful.

    In that spirit, I'd be happy with just capturing the idea of integration tests to start with. Even better if the plugin captures all the different flavors of tests.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • Isn't it just a new task with different includes/excludes?


    test {
    exclude "tests/**/integration/**"
    }

    task intTest(type: Test) {
    include "tests/**/integration/**"
    }

    ?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • It's about more than this.

    One part is about establishing some conventions around integration tests (eg if you put your integration tests under src/test/java in an 'integration' package, then they will be treated as integration tests rather than unit tests). And about defining where in the standard lifecycle these will run. Eg are they run when you do `gradle check`? Or is there another lifecycle task that runs the integration tests and unit tests?

    Another part of this is to add some way to declare 'this project has integration tests', so that other plugins can do useful stuff. For example, perhaps the IDE tasks map the integration tests to a separate IDE project. Or the jetty plugin automatically builds and deploys the web app before running the int tests, injecting the URL to the web app as a system property. Or perhaps the application plugin could build and install the application ready for integration testing.

    As far as implementation goes, it might be better to model integration tests as a separate source set, with separate compile and runtime dependencies, and separate source directory, as it is often the case that unit test and integration tests are fairly independent.
    • >> it might be better to model integration tests as a separate source
      This is the approach we are currently taking with ATG DUST which we are using for container integration tests separate from our standard unit tests. Something like this:

      allprojects {

      sourceSets {
      dustTest {
      java {
      srcDir 'src/dust/java'
      }
      resources {
      srcDir 'src/dust/resources'
      }
      groovy {
      srcDir 'src/dust/groovy'
      }
      }
      ...
      configurations{
      dustTestCompile{
      extendsFrom testCompile
      }
      dustTestRuntime{
      extendsFrom dustTestCompile
      }
      }
      ...
      apply {
      task dustTest(type: Test) {
      description="Runs all ATG DUST tests"
      tasks.dustTest.setGroup('Verification')
      testClassesDir = sourceSets.dustTest.output.classesDir
      classpath = sourceSets.dustTest.runtimeClasspath
      }
      }


      So as dustTest (our integTests) are not build dependent, devs running a build only run unit tests. before checkin they run the dustTest task separately, unless actually creating these integration tests themselves. CI then runs dustTest as a task with a dependency on build ...
    • You might want to take a look at my pull request which is linked to in one of my previous comments. This does pretty much the same thing I guess, but is based on naming conventions and it is possible to create multiple test types and so on. It is also possible to define dependencies, i.e. unit tests are required to run integTest, but you can run unit tests only etc. It eventually went nowhere as the requirements defined by Adam seem to be completely different and defined only after my initial pull request (not complaining, just sayin'), and since then I haven't had the interest and time to do anything about it.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • I'm actually trying to do this now, though I'm not aiming so much for 'unit' vs 'integration' as just plain 'fast' vs 'slow' tests. One batch of our tests runs in about half an hour. Another batch runs in about 3 hours. Yet another batch takes about 8 hours. Just plain 'test' doesn't really cover it. I'm currently considering the (admittedly hacky) method of having testFast, testSlow, and testVerySlow all set variables and then each test{} block in the subprojects have conditionals on those variables, but I'd love some kind of HOWTO or best practices covering this, since I'm new to gradle.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • I’m excited
    I've created a very immature plugin that does the minimum necessary. I needed to add integration tests to my project so I figured I'd try to make something a little reusable. It is hosted publicly here:

    https://bitbucket.org/aispina/gradle-...

    Contributions and feedback welcome!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • I agree with pjimenez3 on simply specifying different include/excludes. For our usage we have different permutations of TestNG groups and java packages that determine what integration tests we run. Fixing this bug - http://issues.gradle.org/browse/GRADL... would allow us to configure this external to our gradle script and drive it from our CI system. I do not like the idea of a build tool forcing you to use a specific source folder hiearchy or package name.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • 1
    I think that we're talking about two different build problems here. Separating tests into groups based on speed is important because developers don't want to wait on a build. Separating tests by function is also important because different functionality often requires a different environment (for example my integration tests need to run against my WAR in a web container, while my unit tests have no external dependencies).

    The problems are related because one way of implementing fast/slow separation is by making separate a source set for each speed. This is the natural approach to splitting by function. However, I agree with Robert and PJ that separate source sets aren't desirable when splitting tests by speed. When splitting by speed, you want to easily be able to mark a test as one or another speed without having to move it on the file system. You also want to be able to support as many speeds as you'd like without having to have an arbitrary number of top level directories.

    This doesn't work well for a division of tests by functionality though. When dividing by functionality, I would expect a user to want to be able to group supporting files with the tests. For example, database files that are only used by integration tests.

    In conclusion, I think there's two problems here that deserve separate treatment. In functional splitting (which my plugin attempts to start addressing) you want separate sourcesets for your tests. In speed splitting, you want a series of include/exclude clauses to define groups of tests.

    I'd like to suggest that we branch this discussion so that the two separate use cases can get the attention they deserve without interfering with each other.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • What is the status on a naming convention for integration test `sourceSets`/`configurations` vs. having a Java package `integration` under the `test` source set. It would be great if we could agree on those namings early on. As integration tests often need to be assembled as Java ARchives first, the separate source set approach seems to be clearer. Also, while camelCase notation works fine when in the gradle script, it should not matter at the filesystem level, in fact it would be better to use lower case only there.

    So what will be the name for the integration test source set ... `integrationTest`, `integrationtest`, `integTest`, `integtest`, `intTest`, `inttest`, `integration`, or something else? Looking at the Gradle 1.1 `org.gradle.build.integtest.IntegTestConvention` class it seems a mix of some of those options.

    Can we please agree on the naming conventions first? Thanks.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • Luke Daley (Gradleware Engineer) August 21, 2012 09:30
    If we had to pick a name for the source set, it would be `integTest`. For no other reason than that is what we use for the Gradle build.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Thanks. What about the fs? Do you agree that cc notation is unusual for folder names?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Luke Daley (Gradleware Engineer) August 21, 2012 12:45
    We haven't had any issues with it in the Gradle project. It should definitely be the same as the name of the sourceSet.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • In hindsight, my integration test plugin is a pretty poor start. Feel free to ignore it. Alas.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Re camel case on integration test folder names: Just remember that in some OSes there cannot be files that just differs in casing.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Re camel case on integration test folder names: Just remember that in some OSes there cannot be files that just differs in casing.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated happy, confident, thankful, excited

  • Hi all. I have been messing around with Gradle recently, and did some initial work on this feature. The repository branch can be found here: https://github.com/wujek-srujek/gradl....
    The first commit contains the core classes that constitute the plugin. The main class, TestingPlugin, contains some ideas that I implemented this with, and also a few questions. This is targeted at Adam especially.
    The second commit contains an example project that uses the plugin and introduces some tests and configures them slightly.
    Feedback welcome.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Here's a rough spec of how I imagined we'd tackle this: https://github.com/gradle/gradle/blob...

    I think an important place to start is to introduce the concept of a test suite. Then, we can add a bunch of different ways of defining and constructing these test suites.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Adam. I have some suggestions for more use cases.
    * I want to measure code coverage for my integration tests
    * I want int test code coverage to be included in sonarRunner
    * In order to shorten the feedback loop, I want to have the possibility to control setup and test invocation so that i can distribute my integration tests towards a number of servers in my machine pool
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • @joachim, good suggestions. I've added the first 2 use cases to the spec. I'm not sure exactly what you meant by the 3rd use case. Can you explain a bit?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • @adam Sure.
    In our product, we have a huge number of integration tests. We also have a policy to include integration tests in the CI feedback loop. So in order to decrease the execution time, we would like to have some kind of testcase distribution mechanism towards a large number of machines. Kind of the same pattern as I believe you use for unit test execution. A number of executors and a distribute or fetch mechanism. The only difference is that for integration tests, it may be distributed to physical machines.
    Does this make my suggestion clearer?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • It does, thanks. You might be interested to know that there is some work happening to allow tests to be distributed over a number of Jenkins slaves.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Regarding the test slaves or machines on which the integration tests run on. In our current implementation we use a plugin and an extension to dynamically book and release the machines via a web service. So Gradle is actually providing the resource name to the deploy and test framework in our environment. And the implementation is a web service plugin.
    But I was thinking of using some other mechanism like Redis (http://redis.io/) so the booking can be done in a standard way.
    Perhaps this is something to consider for the integration test plugin? A 'standard' mechanism to name resources and some other (redis plugin) to provide the implementation of booking and releasing?

    I'll be happy to discuss details of my idea further put maybe off this list.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated