Loading Profile...
Twitter,
Facebook, or email.
EMPLOYEE
summary: A parallel build that does not require the build master or plugin author to concern themselves with writing thread-safe code.
status: Discussions are happening and we will iteratively add features for this after 1.0.
code: planned
Every build has plenty of parallelism which Gradle can exploit to complete the build a quickly as possible. For example, the tests for a particular project can be running while its dependent projects are built and tested. Gradle will provide a general purpose mechanism to spread the execution of a build across multiple processes on a given machine, and across multiple processes running on different machines. Gradle will extract the parallelism of a given build based on information already present in the project model (such as the project and task dependencies, task input and output files, and so on). It will do this without requiring that the build master or plugin author concern themselves with writing thread-safe code.
Distributed execution is not just about speeding up the build. It allows a build to use resources without requiring them to be installed on the local machine. For example, a build user can build multiple native versions of an application without needing to run the build on each operating system. Or, functional tests can run against multiple different target platforms that may not be possible or easy to install locally, such as selenium tests which use multiple versions of Internet explorer. Distributed execution also opens up options to make the build independent of location. For example, a build user might be working at home or at a client site, and can use distributed execution to use a build farm at work, which can access internal repositories and other resources.
There will be very little effort involed in maintaining a build farm. The process for adding a build machine will be simple: Unzip the Gradle distribution and run a script to start the daemon. Optionally, you might specify some meta-data about the build machine. The build master will be able to specify in the build script various constraints on this meta-data which a build machine must meet in order to run the build. Gradle will discover and make use of those machines which are available and meet the criteria. Gradle will offer some capabilities for optionally restarting the affected part of build when a build machine failures. Gradle will expose state and operational capabilities over JMX, to allow centralised management.
Security will be baked into the platform. A build user will be authenticated with the build agent, and the build agent authenticated with the build user. Integration with existing enterprise authentication and authorisation infrastructure will be possible. On platforms where it is available, the builds for each build user will run in their own sandbox. Encrypted transport will be used to move files between the build user’s machine and the build agent.
The Gradle team might also look at being able make use of the build agents provided by various CI tools. We will offer tighter integration with various popular monitoring and provisioning tools.
Parallel and Distributed Execution
-
Have you considered writing a Jenkins plugin that is "Gradle smart"? While fine grain parallelism isn't a good match for Jenkins, some of your tasks for a build farm might be ... -
-
I am waiting for this feature quite a long time. During the builds, a few of the processors of our build server seem to be bored and they should help to reduce the overall build time.
-
-
I’m
excited
Very much like the sound of this, I was wondering whether I could bake my own build parellism using gpars but was very much aware that the risks of gettting things nastily wrong would increase if I manage the parallelism myself. Luke also mentioned that it's not an easy thing to integrate gpars with Gradle at the moment. So to hear that distributed and parallelised builds are on the Gradle roadmap is great! -
-
I think distributed builds is not the right direction for Gradle. As neat as a feature as it would be, I doubt there would be a lot of adoption. Most companies are already tied into a CI infrastructure which they trust to run a server farm. Those CI servers are typically build tool agnostic. I don't see the argument can be made that they would maintain a second server farm that is tied to a single build tool. Likewise, doing anything distributed is HARD, and it would consume a massive amount of developer work which would better be spent on promoting Gradle and increasing adoption.
Now parallel builds on the same machine would be a huge win. Everyone's got CPUs sitting idle, and it doesn't have any of the distribute file system issues, except for a shared cache that should be already taken care of with the new wharf cache. It would fit right into the existing CI ecosystem, while also benefiting developers on their desktop. The task graph seems embarrassingly parallelizable.
One distributed feature which I can see getting adoption is distributed tests. It's a feature that no one else has natively. Which means in the meantime we have to hack it together manually cobbling together groups of tests and running them separately. But even then, long running tests tend to be integration tests which require the developer to consciously make parallelizable since they might be reaching out to a common external system. So, even that isn't a slam dunk. I can barely imagine getting developers capable of running tests in parallel on the same machine, getting them to get their tests working in a distributed fashion is almost unthinkable. I completely understand the argument that tests should be independent, blah, blah, blah. But the world just doesn't work that way sometimes. At a minimum they'd need to be test framework level support. -
EMPLOYEE
* Jenkins Ivy plugin is great at handling upstream/downstream dependencies and module level parallelism across nodes. Unfortunately, it is a Jenkins specific solution, i.e., it doesn't apply to those of us who like to build outside of Jenkins. For that, you have to have separate build logic.
* Gradle multi-module support is great, but has a different DSL to express intra-module dependencies versus the DSL used to declare dependencies obtained via repositories.
I think Gradle needs to have a story that describes how the many people like us can use our existing CI tools with Gradle. My suggestion is to let Jenkins do what it is good at (cron jobs & chunking work across build nodes) where Gradle can be called the same way inside or outside Jenkins using the same dependency DSL.
I'm definitely not complaining. I understand this is a work in progress. I look forward to seeing the result ...