Gradle knows project better than engineer ;). Some time, engineer wants to take control to resume the task where it left instead of gradle always doing it from the beginning.
You can implement this on your own:
def resumeTask = project.property("resume")
if (resumeTask) {
gradle.taskGraph.whenReady { graph ->
def allTasks = graph.allTasks
def resumeIndex = allTasks.findIndexOf { it.path == resumeTask }
if (resumeIndex < 0) throw new GradleException("Can't resume from $resumeTask because no such task is scheduled for execution")
allTasks.subList(0, resumeIndex)*.enabled = false
}
}
Now run with ‘gradle foo -Presume=:bar’.
Note that with the advent of ‘–parallel’ and optimizations that involve dynamic task reordering, this might not work so well anymore.
Thanks for the example, now it skips the upstream task but still takes about 30 seconds to Loading the gradle script and resolving the dependencies. The project had about 500+ tasks and 200+ dependencies. I got the idea and I can tweak it with different option. Thank you very much Peter.
Hard to say from here why it takes so long. ‘gradle someTask --profile’ will give you an HTML report telling you where the bottlenecks are. Resolving dependencies should be extremely quick once Gradle has downloaded them once. What we often see is that builds do something in the configuration phase (i.e. for every single build, no matter which tasks get executed) which they should only do in the execution phase (i.e. when a particular task gets executed).
I agree Peter, it is hard to say what is exactly taking time without looking at the actual script and project structure.
Here is a hiccup in using the script, the resume form work if the task does not depends on the another task so it is not exactly resume from. Let me draw a use case. Suppose if I have issue in the test case and I modified the test case then I have to compile the test case before running it. Since we disabled the compilation it does not quite work for if I resume from “test” or if I resume from compileTestJava then it walks through all the up-to-date check and runs test case on the up stream dependency.
Is there any way we can improve the resume from so above mentioned use case will be covered?
BTW, I came up with another idea to enable task per module by disabling the other tasks. This works great and improves the build efficiency. The script can be enhanced to take comma separated list of project names instead of one project.
To run all the task from one subproject
def subproject = project.hasProperty("subproject") ? project.property("subproject") : null;
if (subproject) {
gradle.taskGraph.whenReady { graph ->
def allTasks = graph.allTasks
allTasks.grep{!it.path.contains(subproject)}*.enabled = false
}
}
Now, I can run gradlew
assemble -Psubproject=:x.y.z
I don’t understand the use case. Why not simple run ‘gradle test’? Using “resume” to speed up the build is, in all likelihood, the wrong solution. Do you know where the bottleneck is? Did you do ‘gradle build --profile’ and inspect the generated profiling report?
The dependency resolution took about, 15 seconds, “All dependencies 14.946s”.
Assume here is my project structure,
project
p1
p2
....
p80
...
p100
and you have modified a test class at project, p80.
‘gradle test -Presume=:p80:test’ – will skip :p80:testClasses and :p80:compileTestJava tasks since it is already in the top of the tree.
‘gradle test -Presume=:p80:compileTestJava’ – will run test cases unnecessary for p1 to p80
The dependency resolution took about, 15 seconds, “All dependencies 14.946s”.
That’s strange. Do you have an explanation for that? Are you using the latest Gradle version?
Anyway, what you should run is ‘gradle :p80:test’. This will just run the tasks that ‘:p80:test’ depends on, nothing else.
The whole point of ‘resume’ is not to run depended-on tasks. It is only applicable in very special cases and definitely not the right tool to improve performance.
Yes, I am using the gradle 1.2 with jdk1.7.0_07. The only explanation I have is number of project is very high.
This dependency resolution is much faster (15 seconds) in my machine since my machine is I7 3rd- generation quad core processor with SATA III SSD disk. This time is very high for another developer who is still using the I5 first generation processor with 720RPM spin disk.
For example… Let’s say we have projects P2 -> P1 that runs tasks T3 -> T2 -> T1. Note: -> means depends on. That is T3 task depends on T2 Task. So the DI graph for execution would be P1-T1
P2-T1
P1-T2
P2-T2 P1-T3 P2-T3
But in reality when P1 has not changed and when we call :p2:t3, the project p1 is verified for any changes (which takes some cpu & disk cycles). This problem gets extrapolated when we have ‘p’ number of projects and ‘t’ number of custom tasks (pt number of effective executable tasks and pt number of up-to-date-checks when no project has been changed). In my setup, p=100 & t=10. One of my most frequent use case is to run ‘t’ tasks for 1-3 projects, which unfortunately requires up-to-date check for all the 97 projects which i know for sure has never been touched. In the case of maven, “-rf” option can be used effectively to ignore the 97 projects (up steam dependency) without even running up-to-date-checks. Is there a way, I could do the same using gradle On a side note, I understand that ‘resume’ is not a viable option as it does not do a ‘smart ignore’ on up-stream dependency. But would be potentially useful for other situations.
On a side note… The reason for my performance concern on up-to-date check is becoz, I run my project on an ‘enterprise-IT-managed-machine’ that uses disk encryption, anti-virus plus other goodies that can make sure my build can slow down :(.
First thing is to make sure that you aren’t doing unnecessary work in the configuration phase of the build, because this work will be done for each and every build.
If this doesn’t solve the problem, one way to speed up the build is to script ‘settings.gradle’ to only include a subset of projects, and treat the other projects as binary dependencies. This will require some development effort on your side, though. A future Gradle version will provide this out-of-the-box.
Another option is to adapt ‘resume’ to resume from the first task of a particular project. Then your project will not only get tested but also compiled.
I happy to hear that this option is considered in the future version of gradle.
As I mentioned starting from compileTestJava unnecessarily runs test cases from upstream dependency project. This option also does not help.
Thank you for all your help and I will look forward for the out of box feature from gradle.
As I mentioned starting from compileTestJava unnecessarily runs test cases from upstream dependency project. This option also does not help.
Sounds like something is wrong with your ‘resume’ code, then.
I have same exact code it is avaialble in the first reply. The task graph will be look like
:p1:compileJava :p2:compileJava … :p100:compileJava … :p1:compileTestJava :p2:compileTestJava … :p80:compileTestJava … :p100:compileTestJava … :p1:test :p2:test … :p80:test :p100:test
If I start on the :p80:compileTestJava then it compiles p80’s test code and it runs all the test cases from p1 through p79 but I am interested only :p80:compileTestJava and :p80:test.
For this reason, enable task per project is doable way as of now.
Are you running ‘gradle test’ from the top directory? This will run all test tasks. Of course, you should only run the test task for p80, either by cd’ing into its project directory, or with ‘gradle :p80:test’. Then no other ‘compileTestJava’ task should get executed.
Yes. I am running from top level directory.
‘gradle :p80:test’ – will run all up-to-date check which we want to avoid. ‘gradle :p80:test -Presume=:p80:compileTestJava’ – will work. this is similar to subproject option, isn’t it? It is not resume from since we are going to miss all the task from :p81 onwards.
I can draw a use case. Time T1: ‘gradle build’ – failed at :p80’s test case Time T2: User updating the test case Time T3: you have three option
-
‘gradle build -Presume=:p80:test’ – it does not compile test case so it is not a preferred option.
-
‘gradle build -Presume=:p80:compileTestJava’ – it is going to run all the up-stream dependency test, which is not required.
-
‘gradle :p80:build -Presume=:p80:compileJava’ – it resolves the current one but does not continue :p81 onwards
Peter, you are providing so much help and my hat’s off to you.
I see. So the aim is to have a snappy way to resume a full build after fixing a problem in a particular subproject. I’d say ‘gradle build -Presume=p80’ is the best compromise then, along with a slightly modified implementation of ‘resume’ that disables all tasks prior to the first task that belongs to p80.
Yes, You got it. It is similar to -rf option in maven.
I assume you already did this? It’s just a small modification to my initial code.
I did not do any modification to your code. Can you tell what modification that I should do?