Gradle hangs indefinitely in testClasses (CI)

On our CI environment, running our Gradle build hangs indefinitely on the testClasses task. The job is at http://ci.hibernate.org/job/hibernate-orm-master-h2

The job is configured to run ‘./gradlew clean test check :documentation:aggregateJavadocs publish’

I have tried a number of different Oracle JDK (7) versions to rule out JDK problems, including 7u21, 7u45, 7u51. No difference.

For one run I enabled strace to try to better see where it hangs. Unfortunately the yielded a 3.4G log file that is not very useful due to its sheer size.

I have also tried running Gradle with --debug and --info options. The problem is that using either of those causes the hang to not happen.

At this point I am completely out of ideas, but still the Gradle build hangs.

NOTE : I say it hangs in testClasses because that is the last line I see in the Jenkins build console:

:hibernate-core:runTestSourceGenerators UP-TO-DATE
:hibernate-core:compileTestJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
  :hibernate-core:processTestResources
:hibernate-core:testClasses

Maybe it actually hangs trying to transition to the next task.

Can you provide a thread dump?

How would I get a thread dump from a running Jenkins job?

In TeamCity you’d just press a button. Perhaps Jenkins offers something similar? An alternative would be to log in to the slave machine and run ‘jstack’.

Also, which process? I generally see GradleWrapper and GradleWorker

Should be ‘GradleWrapperMain’.

Is there a particular process/thread you want me to paste here? Or should I paste the whole dump?

Right I had to use jstack. Jenkins does have a thread dumpl plugin but we have not installed it as of yet

GradleWrapperMain process, all threads. Please make it a GitHub Gist or something, and paste the link.

That’s just asking for broken links later when people find these threads. But ok, whatever

https://gist.github.com/sebersole/9979921

Thanks. Looks like the build is stuck during test execution. Can you also provide a dump of any ‘GradleWorkerMain’ process? Meanwhile, you could try to vary ‘tasks.withType(Test) { maxParallelForks = …; forkEvery = … }’ and see if this changes anything. Which Gradle version are you using?

MMM, then why is there not a single reference to “org.hibernate” anywhere in any of these traces? Both dumps are there, scroll down

Looks like it’s stuck trying to shut down one of the test worker processes. Not sure why…

What do I put for the “…”? I am using 1.9

e.g. ‘1’ and ‘100’ (in that order). Aren’t you setting these properties already?

no, i dont touch these settings at all

Is there anyway to specify these from command line so I can try that out on CI directly first?

Not without touching the build script.

So setting maxParallelForks=1 and forkEvery=100 this now hangs locally on my laptop…