Revolutionary new Gradle Features on the 2014 Roadmap

We are convinced that Gradle 2.0 is the best available enterprise build system. Yet we are far from done. Finally, we have the R&D bandwidth to make deep improvements to Gradle in some areas where it is still lacking. We also have the bandwidth to make some fundamental innovations in the domain of build and continuous delivery. These things will bring Gradle much closer to our vision of the ultimate build system. With the improvements noted below Gradle will revolutionize build system performance and extensibility, multi-platform dependency management, IDE integration and native builds. Work on this has commenced and significant portions will be finished by the end of the year. As per usual, all changes are happening on the master branch and shipped incrementally via our regular Gradle releases, every 6-8 weeks.

Performance

There is no single metric that can describe the performance of a build system. Build performance has many facets: small or large project, what command you are executing, is it a clean or incremental build, etc … All aspects of build performance are of utmost importance to us. Since the Gradle 1.0 release we have tremendously improved the performance of Gradle in almost all relevant areas. Gradle shines in many aspects of build performance. Gradle has for example introduced incremental builds into the Java world. But there is enormous potential for further improvements.

Improving the performance will be a major focus for the next 6 months. What is good will become even better and where there are bottlenecks they will be removed. We are talking about revolutionary performance improvements that will make Gradle an extremely fast build system for all types of projects.Result of that will be lightning fast Android Studio Gradle integration and dramatic speed improvements for the largest enterprise builds in the industry like the Gradle build at LinkedIn with over 4000 subprojects. Our roadmap for performance improvements continues into 2015 with plans for fully distributed builds.

Here is what we are going to do in the area of performance:

Much faster configuration time

Right now there is a trade off between the richness and extendability of the Gradle build language and the time it takes to configure the build. The way Gradle configures a build has not changed since its inception. Every time you execute a Gradle build everything is configured, regardless of what command you are finally going to execute. During this configuration phase a build model is constructed, but it is not persisted. This means the configuration has to be re-run every time a Gradle build is executed. Obviously this is wasteful and for larger projects this can mean a significant delay before the actual build execution starts.

This configuration delay is also painful when it comes to IDE integration. Android Studio demonstrates the enormous potential of a deep integration between the IDE and Gradle. At the moment, the price we pay for this is often mediocre responsiveness: every time the IDE makes a query of Gradle, the configuration phase is run before an answer can be given. The folks at Google are implementing some smart workarounds for this. But what is ultimately needed is a fundamental solution within Gradle that eliminates the overhead in the IDE-to-Gradle communication.

Gradle already has a mechanism to alleviate the configuration time problem somewhat for multi-module builds: configuration-on-demand. And the Gradle daemon also caches certain aspects of the model. But this is not a complete solution to the problem.

Configuration Tasks

Currently, the configuration phase runs a bunch of scripts against the Gradle Model objects in order to set up the executional model. What we are working on right now is removing the concept of a distinct configuration phase altogether, resulting in a single execution phase. Configuration will be modelled as tasks which have inputs and outputs like normal Gradle tasks. These “tasks” may not create a jar but instead create a classpath or a library definition. In practice, this will mean:

  • Only configuration logic that is required for the current build command will be executed - All configuration outputs will be persisted. So when build description doesn’t change, the configuration model can be read from cache. Only when one of the configuration inputs change will the outputs need to be rebuilt. See also the section Caching and Sharing Everything. . - In cases where configuration does need to be rebuilt, this building will be both incremental and parallelizable. See also below the section Fundamentally Parallelizable.

These changes will dramatically reduce or in many cases even eliminate the configuration time for Gradle builds. Expect to see the first results in the coming months.

The new model for build configuration is not driven simply by performance requirements. It is more intuitive and logically simpler than the current model. We expect many additional benefits to follow from this innovation, some of which are discussed below.

Fundamentally Parallelizable

Gradle offers already parallel builds. Many of our users make use of this feature on a daily basis. But there are limitations:

  • Granularity of parallelization is per project. - Users must ensure that projects are properly decoupled, allowing them to be built independently. - The configuration phase for projects is not run in parallel.

The improvements that will make the configuration time much faster will also pave the way for making Gradle fundamentally parallelizable. By providing Gradle with deep knowledge about the interactions between different elements of the build, this will enable Gradle to easily and reliably parallelize on the task level without requiring the build author to care about decoupling. This means that parallel Gradle builds will be easier to maintain and the parallelization will be more fine grained. This will provide in general a much better CPU utilization. In a complex build domain like Android (or C++) where you build many variants of a component within one project, task level parallelism will significantly speed up building even for smaller projects. Parallel execution will be eventually switched on by default. In 2015 we will work on making parallelization even more fine-grained than the task level.

Caching and Sharing Everything

Gradle already does a great job when it comes to caching build outputs and building incrementally. We want to push this two steps further. Currently build outputs are cached locally per build, so the first step for us is to have a single cache for the build outputs for all builds on a machine, similar to how we cache external dependencies. This will mean that outputs can be reused across different builds, which will be particularly helpful on CI machines. As a next step this cache will become distributed, allowing any build output to be shared across machines. For large enterprise builds this will mean further significant performance improvements. Cached outputs will include not only traditional task outputs, but also the outputs of the new configuration tasks.

The base concept for such a cache is that every task is a deterministic function from input to output, with the input including all relevant parameters. In the case of compilation for example, the input is not only the source code but also the version and the type of compiler, and the OS being used. When all of those parameters are captured, the cache output can be reliably shared across machines. Having a single task cache is something we plan to commence this year, continuing work into 2015. The internal Google Blaze build system introduced the concept of such a cache years ago. We are glad that Gradle will be able to offer it’s own version of this very powerful concept.

Dependency Management

Gradle has already the most powerful dependency management in the industry. We are now taking this one step further:

Variants

To this point, dependencies have been described via their group, name and version as fundamental properties. We don’t think this is enough. On every major platform you will find additional fundamental properties that should be part of the resolution process. In Java it may be the Java version a certain library is build for. In Scala and Groovy it may be the Groovy or Scala version used to compile the library, as those languages are not necessarily backwards compatible across compiler versions. In Android and C++ things get even more complex with variants built for different CPUs, operating systems, app stores, etc … JavaScript brings also very interesting requirements to the mix. These problems have not been solved by existing dependency management systems. For example different variants of a dependency may have different transitive subgraphs. They may need compatibility rules defining which variants may be replaced with which other, etc…

Work has started on making the concept of variants a first class citizen for Gradle dependency management. For the native world and for Android, this support is a core requirement permitting true dependency management. For any other domain it will be a very valuable addition.

Other areas

Besides variants, there are other aspects we are planning to improve in the area of dependency management. We want to make arbitrary custom metadata a first class citizen in dependency resolution. We also want to provide better solutions for many use cases that currently rely on snapshot versions. This work is not yet scheduled, however the Gradle community has started to tackle some of them already. See for example the Netflix Nebula Gradle Dependency Locker plugin and Publishing Plugin.

Rich Extendability

One of the original contributions of Gradle to the build domain is its declarative and yet flexible build description. A build description that also comes with a sane and rich extendability model. We don’t know any other build system that provides something similar. This model is for example a major reason that Google choose Gradle as the new official build system for Android. We want to take this quality of Gradle to the next level as part of our work to improve the way Gradle builds are configured.

Over time, the Gradle plugin ecosystem is become broader and deeper. More and more we see rich transitive relationships between plugins, with one plugin extending another. The Google Android plugin provides the base build framework for Android. Many other Gradle plugins build on top of the Android plugin to extend the Android build ecosystem (e.g. plugins from various Android cloud testing providers). Next, a particular organization may need to further extend and customise the behaviour of those plugins. Although Gradle makes such scenarios possible by providing many hooks, we have reached the point where this solution does not have the elegance and expressiveness we would expect.

The new configuration model will make it easier to deal properly with complex configuration time ordering issues. But we won’t stop there. Gradle domain objects will be first class citizens within the new configuration model. For example a SourceSet will know exactly which configuration tasks will affect its state. You will be able to hook in custom configuration tasks directly with the domain object, for example enforcing that your custom configuration of the SourceSet should happen after all other configuration of the SourceSet object has been taking place.

C/C++ Support

Gradle has already excellent C/C++ support, and we will continue to improve this. We want to turn Gradle into the most powerful native build system on the planet. The new configuration model will add capabilities that are particularly helpful for the complex C/C++ domain. The dependency management work will make Gradle the first true C/C++ binary dependency management solution. And work on performance is particularly crucial for the adoption within pure native stacks.

We are excited that the Google Android team is currently working on moving their NDK support from Make to the Gradle C/C++ plugins. Once released, this will immediately turn Gradle into one of the most widely used C/C++ build systems in the world.

Tooling and IDE support

With it’s tight Gradle integration, Android Studio is pioneering how modern IDE’s should integrate with build systems. Gradle enables this kind of deep IDE integration with the Gradle Tooling API, which provides a mechanism for programmatically interacting with Gradle and lies at the heart of this effort on the Gradle side. We continually add to these capabilities in cooperation with the Google Android team, for example by enabling rich test execution from the IDE via Gradle. Later this year, Gradleware will offer a Gradle Eclipse plugin which will provide outstanding Gradle integration with Eclipse.

Daemon with Watcher Mode

In partnership with LinkedIn we will add a watcher mode to Gradle, making it possible to keep a Gradle task continuously up-to-date in the background. Whenever the input for the task changes, the task will automatically be executed. We’re really looking forward to bringing this feature to Gradle, which will prove useful with authoring tools like Asciidoctor, JavaScript application development, Scala Play and countless other use cases.

Compatibility

We take backwards compatibility very seriously. We will continue to support the current configuration model for a long time, and we will provide a way to map the current DSL to the new configuration model. However, the benefits of the new configuration model will be so outstanding that we expect many people will migrate towards it very quickly, and we will try to make this as painless as possible.

Final Words

Software code bases are growing in size and complexity. Teams must deal with a mix of different languages and technologies, navigate the build and release challenges of a micro service architectures, and deal with expectations of continuous delivery.

The build system must enable a continuous integration process for such software stacks, by connecting all of the different modules together and giving fast feedback on the correctness of any change. Without such a build infrastructure (and most larger projects don’t have this) the productivity of the software development process suffers terribly. The mission of Gradle is to provide a tightly integrated and super fast build infrastructure for even the largest and most complex software stacks on the planet. At the same time it should be approachable for smaller and simpler projects. The features above are a massive step towards those goals.

1 Like

Awesome to hear that your a working on improved Gradle support for the Eclipse IDE. I think the Springsource tooling is already great and they also working on improvements, see for example https://issuetracker.springsource.com/browse/STS-3659.

Would be great to see a cooperation for Gradeware and Springsource on the Gradle support in Eclipse.

Good point. We are coordinating our efforts with the folks from SpringSource.

Something missing from core Gradle is the concept of releasing. Currently the best release plugin (townsfolk) suffers from proper integration with other plugins. When versions are bumped or a -SNAPSHOT removed, other tasks like Jar and Publish get confused. Since the concept of cutting a release is so fundamental, I think it would be great if the Gradle developers took over the development of the release plugin to ensure it works properly with the other core features.

I concur heartedly!

Without a doubt covering the complete development lifecycle should be the most important activity. Creating and publishing artifacts to repositories (in my case nexus repos) with automatic version numbering is critically important for inclusion into continuous integration environments.

Thanks for your replies. Two things I would like to say:

We are on the same page that what you are describing is very important functionality. We have a lot of ideas what we can do there. We are also doing already some critical foundation work in the area of dependency management to enable very good solutions for releasing/staging (e.g. allow any custom metadata to be added and used for dependency resolution). The community is also adding valuable tools in this area, see the https://github.com/nebula-plugins/nebula-publishing-plugin which deals with Maven snapshots when publishing. As you said, eventually we want to have a core, coherent, full development lifecycle solution for this. No question.

Whether this is THE most important thing to do is not as easy to answer. We segment features in four buckets. One is platform support (e.g. Java, Scala, JavaScript, …). Another is continuous delivery lifecycle (the features you are mentioning). The third is scalability. The fourth is Gradle core.

I can completely see that the features you mentioned are the features you are missing most. And you won’t be alone with that. But for other parts of the community the major pain lies somewhere else. For us it is super important that IDE integration works better and much faster. For Android Studio, and we have many, many Android users, this is the biggest pain. For larger Gradle builds configuration time is also a significant pain. On top, Gradle platform support providers like the Android team, need stronger configuration capabilities. As they are working on getting to a stable 1.0 release as soon as possible, we want to add all the features necessary to do this in a way that the solutions are very good. Our native plugins require similar capabilities. Therefore we decided to have a burst of work on the buckets 2-4 as it will solve so many important problems at once. As those changes will also affect how people are writing Gradle build scripts, that is another reason why we want to have those changes as soon as possible.

We can’t wait to dedicate more time to improve the capabilities of Gradle to model the whole continuous delivery cycle in more depth. This will also include deeper support for typical integration testing scenarios and a deep modeling of service dependencies on top of the functionality you have mentioned.

In the meantime, if you can spec out your requirements in more detail, that would be much appreciated input.