Over the past couple of weeks we’ve made some significant improvements to the way dependency resolution works in Gradle. There are some things that are just refactorings, some performance improvements, and a couple of new features. These changes should make it into M6, which is due out soon.
Caching Improvements: -------------------- A big change is that Gradle now takes control of the caching of Dynamic Versions (eg 1.+) and Changing Modules (eg SNAPSHOT). A new metadata file in the cache, “dynamic-versions.bin”, caches information about module resolution on a per-resolver basis.
Dynamic Versions:
If a resolver tells us that 1.+ is actually 1.5, we remember that for next time. That way, we can look directly in the module cache to get the descriptor for 1.+, rather than first having to resolve 1.+ to 1.5.
Dynamic Version cache entries expire by default in 24 hours. This value can be modified using
‘configuration.resolutionStrategy.cacheDynamicVersionsFor 2, “minutes”’
Changing Modules: Similarly, if a dependency is flagged as changing, or a resolver tells us the module is changing (eg SNAPSHOT), we remember when we last fetched the module meta-data and artifacts. If we find a cache entry indicating it’s ok to use the cached version, we use them. Otherwise, we bypass the cache and download the descriptor and artifacts again.
Changing Module cache entries expire by default in 24 hours. This value can be modified using ‘configuration.resolutionStrategy.cacheChangingModulesFor 2, “minutes”’
Maven Snapshots: By implementing these resolution caches ourselves, we have been able to remove the previous snapshot ttl that was provided by the WharfIbiblioResolver. This has greatly simplified things internally.
Offline Mode: Although nothing official has been added, it should be possible to add an offline switch to your build that sets these cache values very high. In theory, no requests should be required when everything has be previously retrieved. An official way to run your build offline is planned for 1.0.
You can read more about the support for Dynamic Versions and Changing Modules in the docs.
Performance Improvements: ------------------------ Although the new caching improvements help a lot with performance when things have been resolved before, we’ve also done some work to improve the performance when resolving dependencies against maven repositories. Namely:
- Don’t request javadoc and source jars when resolving the POM - Don’t request .sha1 and .md5 files when not required - Don’t use HTTP HEAD to check existence of file prior to downloading them.
Together, these improvements have reduced the number of HTTP requests required to resolve a static module from 11 down to 2, and a simple SNAPSHOT module from 25 requests down to 4!
Other Refactoring: ------------- We spent a bunch of time replacing the core Ivy ResolveEngine with our own implementation. This has already proven beneficial in making the other improvements simpler.
We’ve also started using our own from-scratch implementation of org.apache.ivy.plugins.resolver.DependencyResolver, in place of extending AbstractResolver, BasicResolver, ChainResolver etc. Our own implementations need only implement 3 key methods on the interface, and removing a whole bunch of unnecessary (and misunderstood) complexity in the process.
Another improvement has been to remove some of the power of the low-level ivy resolvers that perform the actual resolution. Previously, each resolver could look at what had been resolved earlier in the chain, and determine if it do anything or accept the earlier resolution. Now each resolver behaves independently, with a higher level controller doing the work of choosing the best resolved version, short-circuiting resolution when looking up static versions.