Loading Profile...
Twitter,
Facebook, or email.
EMPLOYEE
summary: Making it easier to retrieve, publish, host, test and document plugins and providing an online service for hosting Gradle open source plugins.
status: Discussions are happening and work will start not long after the release of 1.0
code: planned
The plugin portal will enable Gradle users to more conveniently reuse plugins and scripts developed by other Gradle users. The portal will act as a database, with an interface to search for and find out about the plugins that are out there. Other planned features:
* Hosts multiple versions of the same plugin
* Runs nightly automated tests to validate which versions of a plugin works with which version of Gradle (including the latest Gradle from trunk).
* Provides a staging workflow for publishing new plugins (Do they provide documentation, integration tests, …)
* Gradle would also provide a Plugin Development plugin that makes it very easy to write and publish plugins.
* The source and the actual binaries of the plugins can live wherever the author wants them to live.
This will include a set of Gradle plugins that make it easier to retrieve, publish, host, test and document plugins. This won't depend on the host which might be the official Gradle open source plugin portal but can also be any internal enterprise repository manager.
Plugin Portal & Improving Plugin Development
-
1This is at the top of my personal wish list for Gradle. These are my primary pain points: "...retrieve, publish, host ...".
I'm keen to work on these issues and contribute improvements. Finding the best way to fit into the roadmap and agree on the correct direction for Gradle have been the main blockers to date.
GRADLE-1677 was a step towards the retrieve portion, as was GRADLE-1653. These two mailing list discussion threads relate to GRADLE-1677:
http://gradle.1045684.n5.nabble.com/e...
http://gradle.1045684.n5.nabble.com/I...Comment -
-
I'd like to revive the discussion around the retrieval part of this roadmap item. Is this forum topic a good place for it?
-
-
yes it is
-
-
@adam Great!
Here's my recap of this thread
http://gradle.1045684.n5.nabble.com/e...
In terms of the user experience, we settled on this:
"Being able to inline the dependency declaration into the apply statement is something we've discussed before. The plan is to have the apply statement handle dependency coordinates. You would be able to do any of the following:
1. apply group: 'mygroup', name: 'myplugin-project', version: '1.0', plugin: 'myplugin'
2. apply group: 'mygroup', plugin: 'myplugin', version: '1.0'
3. apply plugin: 'myplugin'"
We then came up with a technical solution:
"I think there are really 2 options to solving this:
1. Limit the places where this type of apply statement can appear in the script, eg it must be a top-level statement.
2. Don't even try to support changing the build script classpath.
I'd go with option 2. The apply method would work something like this:
1. create a new configuration containing the plugin dependency, and resolve the configuration.
2. create a URLClassLoader from the resulting files. The parent ClassLoader should be GradleInternal.getScriptClassLoader().
3. use the ClassLoader to look up the plugin.
Nice and simple. It might be good to have a cache of classloaders created in step 2, so we can share the implementation classes across all projects in a build (and, when using the daemon, across builds).
What is interesting about this approach is that plugins are isolated from each other. The big downside is that the plugin cannot contribute classes to the build script. But I think it might be a good thing to move away from needing to do this."
During the discussion of related issues to the technical solution we stalled here:
"The more I consider this, the more it just feels wrong (by “this” I mean not exposing types from plugins to the build file compile). I can see it being a source of much confusion and annoyance to users. Not having types as first class citizens has an unclean feel to it and seems like a compromise which has the potential to turn people off. I think we need to keep searching for a cleaner solution."
What would help us proceed? -
-
Something that caught my eye from the other thread was the wish to be able to do something akin to this also:
apply plugin: 'company-plugin', fromJar: 'url://to/company-plugin.jar'
See: http://gradle.1045684.n5.nabble.com/I...
Although for my taste anything we do needs to be compatible with working without a network connection. -
-
I’m
happy
I had suggested that, but I think I can live with using apply from.
This seems to work for me:
Remote Script:
buildscript {
repositories {
mavenRepo urls: ['http://internal.company.com/artifactory/repo']
}
dependencies {
classpath 'com.company:company-plugin:11.4'
}
}
apply plugin: 'company-plugin'
Local Script (Single Project):
apply from: 'file:///Users/elberry/development/projects/mine/test/remote.gradle'
Local Script (Multiple Project):
subprojects {
apply from: 'file:///Users/elberry/development/projects/mine/test/remote.gradle'
}
This seems to solve my issues, and seems to work fine on 1.0M3. I think there were some drawbacks to this though, and I haven't tested it with 1.0M5 yet. -
-
I’m
frustrated
This no longer seems to work for me in milestone-5.
There is an existing bug for this however: GRADLE-1517 -
-
I’m
excited
Wouldn't it be best if there was a central repository hosted by gradle where all the plugins could live. This would make using them much easier in the long run since we would not have to track the location of each plugin, but can simply rely on the central repository.
- view 1 more comment
-
-
We use artifactory to proxy external repositories to protect us against outages and to increase speed for the team.
If the plugins where not in a central maven compatible repository but spread out over the net we could not proxy them with artifactory. -
-
My intention for this is that if we do allow you to host your binaries anywhere, we will proxy it in some fashion so there is always a consistent URL scheme you can hit.
My preference at this stage is this approach, but at least initially it may be a requirement that we host the binaries. -
-
-
-
I'm looking to work on this again next week (GRADLE-1677) at the local Hackergarten event. Are there any high-level decisions we need to make beforehand?
Quick recap:
"Being able to inline the dependency declaration into the apply statement is something we've discussed before. The plan is to have the apply statement handle dependency coordinates. You would be able to do any of the following:
1. apply group: 'mygroup', name: 'myplugin-project', version: '1.0', plugin: 'myplugin'
2. apply group: 'mygroup', plugin: 'myplugin', version: '1.0'
3. apply plugin: 'myplugin'
"-
It would be nice to have a short notation, e.g. _apply plugin: "mygroup:myplugin-project:1.0:myplugin"_ or _apply plugin: "myplugin", from: "mygroup:myplugin-project:1.0"_
-
-
and perhaps a shorthand for specifying repository along with it. Eg.
apply plugin: "mygroup:myplugin-project:1.0", mvn: "http://host/repo"
or
apply plugin: "mygroup:myplugin-project:1.0", ivy: "http://host/repo/[artifact]-[pattern]"
-
-
-
-
-
@Merlyn: the build classpath needs to be constructed before any calls to apply(), so it's not clear to me how we'd use the dependency coordinates so late in the lifecycle.
Apart from that problem, I like the notation. -
-
Yes. That's tricky. Either the apply function needs to be part of the first pass of the script (tricky/dangerous), or the dependency of the plugin is not shared (with the build script itself and the other plugins).
Last time we discussed this the sticking point was around making classes like Tasks available to the build script.-
Treating these things like script plugins in that they'll have their own isolated classloader hierarchy sounds like a good idea to start with. That should work.
-
-
-
-
-
I can see 4 broad strategies here:
1. We extract all the calls to `apply` in a pass that runs before we assemble the final script classpath, and use these calls to decide what to add to the script classpath. This may or may not be the same pass that evaluates the `buildscript { }` blocks.
2. We use some mechanism for adding stuff to the script classpath that is separate from applying the plugin (this is the strategy we use now).
3. We get rid of the need for plugins to be visible on the script classpath.
4. Transform the build script so that unresolved static types are replaced with dynamic types, and have the `apply` statements contribute stuff to the script classpath dynamically as they execute.
1) is just too complicated to get right, except perhaps for top-level `apply` statements early in the script (which they often are). So we could extract only a certain subset of `apply` statements, for example, perhaps these are ok:
apply plugin: "group:module:version:plugin"
subprojects {
apply plugin: "group:module:version:other-plugin"
}
2) doesn't necessarily have to be as complex as it is now. For example, perhaps `apply` statements in the `buildscript { }` block contribute to the script classpath, and those outside the `buildscript { }` block do not:
buildscript {
apply from: "group:module:version:plugin"
subprojects {
apply from: "group:module:version:other-plugin"
}
}
However, to me, 3) is the way to go, perhaps combined with 2) for those cases where it really does make sense.
So, given this, I would have `apply` create an isolated classloader for the plugin, with some caching so that we reuse the classloader across build scripts. -
-
Thanks, Adam.
So if we go with 3), how do we make plugin Types available to the build script? E.g. the Xslt task: http://wiki.gradle.org/display/GRADLE...
Would that be a scenario in which we would use 2) ?-
The plugin would register the type somewhere. For example, it might, by convention, make the type available through its extension, so that you do:
apply plugin: 'myplugin', dependency: 'some:dep'
task customTask(type: myplugin.Custom)
where `Custom` is a property of type `Class`.
We might even make this happen automatically, so that the plugin author does not have to register the type. For example, when you access an unknown property of an extension, we might look for a class with the given name in the plugin's classpath. For example, `myplugin.Custom` looks for a class with simple name `Custom` in the plugin's classpath without the plugin author having to do anything. -
-
-
-
-
Hi all,
Luke Daley and I worked on a solution to make applying plugin dependencies more concise. This is the syntax we settled on:
apply plugin:'pluginId', dependency:'group:artifact:version'
This works by performing a transformation in the first pass which converts the apply call into a call on the ScriptHandler. The second pass then applies the plugin as usual. This solution means that classpath additions are accessible to the build script.
The code isn't ready for a pull request. It is however ready for criticism as regards the overall approach. Would you take a look at it?
The commit is here: https://github.com/curious-attempt-bu...
The integration test is here: https://github.com/curious-attempt-bu...
Cheers,
Merlyn -
-
I'm reluctant to go with option 1). I think you're going to have difficulty making it work well. What's your plan for dealing with some of these cases:
subprojects {
apply plugin: 'someplugin', dependency: 'some:dep'
}
libraries = [somedep: 'some:dep:1.0']
apply plugin: 'someplugin', dependency: libraries.somedep
groovyProjects = allprojects.findAll { // ... }
groovyProject.each { it.apply plugin: 'someplugin', dependency: 'some:dep' }
class SomeType {
void apply(Map) { }
}
configure(new SomeType()) { apply dependency: 'this-is-not-an-apply-statement' }
pluginVersion = '1.2'
dependsOnChildren() // child project mutates pluginVersion
apply plugin: 'some-plugin', dependency: "some:dep:$pluginVersion"
def configureProject(String dependency) {
apply plugin: 'some-plugin', dependency: dependency
}
configureProject('some:dep:1.2')
I don't think you can solve this generally with static inspection. You're going to have to restrict the cases where the classpath is mutated to a subset that you can statically determine.
I'd rather we went with option 3). It will require a little more effort on the part of the plugin author (which we can address in other ways), without the cost of the complexity that the build script author has to deal with. Or the complexity of trying to deal with classpath conflicts between the build script and the various plugins it happens to require. -
-
@Adam: good points.
It's starting to look like this isn't going to be practical to solve.
An implication of using indirection for Class object references is going to be that it's harder for IDEs to provide intellisense in build scripts.-
Not really, if we're doing the registration declaratively. The idea is that we (via the tooling API) provide the ide with information about the type of a given identifier, and which identifiers are available. If this content assistance is based on static analysis, it's not going to be any different whether the classes end up on the script classpath or not. And same for if the content asistance is based on some runtime evaluation. Either way, a given identifier has-a runtime type and a given type has-a implementation classpath.
-
-
The difference would be that types would need to be imported, and IDEs could leverage that to get some info. But, that's probably not that valuable as it can't address everything.
Also, if our long term strategy is to provide rich information via the tooling API then it seems like we should more or less give up on trying to put these things on the classpath proper as suggested. -
-
-
-
-
Okay. So tweaking the build script classpath is out. Instead I presume we create a classloader for each plugin.
So here's a idea from a different direction, that Luke and I came up with. How about a naming service that can resolve a pluginid (plus any other relevant information such as the gradleApi version) to a way to resolve the plugin artifact and dependencies?
That way as long as the naming service in use knows what "tomcat" maps to, the script author can write:
apply plugin: 'tomcat'
The naming service could be a simple restful service, and the Gradle-blessed naming service could either be included by default, or easily available via a method e.g.
namingService { gradleNamingService() }- view 2 more comments
-
-
-
> The functional difference between a plugin loaded this way and by the traditional seem too subtle and intricate too me and will ultimately confuse.
I can only see 2 differences:
* You can't import the types from the plugin.
* You won't run into the weirdness of trying to deal with conflicts between your script classpath and those of the plugins you use, as now they're all isolated from each other.
This model is exactly the same as that used by apply from: 'script', so there's some goodness we pick up from the consistency there.
> It's also not really practical without infrastructure for hoisting class objects up to the buildscript.
Of course it is. Many builds don't instantiate any tasks from the plugins they use, they simply configure the tasks or domain objects that the plugin adds.
We don't necessarily need any infrastructure for people to get started. People have been using this model with applied scripts for quite a while now.
Having said that, perhaps a good way to tackle this problem is to first add some (basic) infrastructure that can work for applied scripts, then add the new apply method which can then make use of this infrastructure, then grow out the infrastructure. -
-
> Of course it is. Many builds don't instantiate any tasks from the plugins they use, they simply configure the tasks or domain objects that the plugin adds.
Granted, but it does happen and it's something we support / promote. Or at least, we don't discourage it.
> We don't necessarily need any infrastructure for people to get started. People have been using this model with applied scripts for quite a while now.
Sharing of 3rd party scripts is far less common. I can't think of any widely used community plugins that are distributed via scripts. My concern is that the picture for loading plugins/build time dependencies is already non trivial to understand and we are adding more complexity here. Sure, the syntax is getting more concise/simpler but in terms of functionality we are introducing something new.
I'm not against the idea of leaving the compile classpath loader alone as I think it's the only real solution given your points. Given that this is the direction we are going, perhaps we should just add this without anything else and deal with any support fallout that might occur because it's somewhat temporary. If you're confident that we'll eventually have robust solutions for dealing with types introduced by plugins loaded in this manner in a reasonable timeframe I'll withdraw my objections. -
-
-
-
I like this idea. It is pretty much independent of how we choose to deal (or not) with the classpath, so we can do this regardless of whether we go with option 1) or option 3), or some other option wrt to classloading.
Some things we need to think about:
One thing that using a dependency definition in the apply statement gives you is namespacing of the plugin ids, ie you're giving a (group, module, version, plugin-id) tuple which nicely namespaces the plugin. By just supplying an id, this becomes potentially ambiguous. And this ambiguity can change over time, as the set of available plugins grows, so you have problems with reproducibility.
On a similar note, using a dependency definition automatically gives you versioning of the plugin. If you just supply an id, then the version becomes ambiguous. And the ambiguity can change over time.
My thought here is that we model things like this:
* A plugin has a composite id (group, plugin, version). This is similar to, but not the same as, a dependency module id.
* A plugin has an implementation classpath.
* There are 3 steps to applying a plugin:
* Determine the plugin id from the parameters provided to the apply statement
* Given a plugin id, determine the implementation classpath.
* Given an implementation classpath, instantiate and apply the plugin object.
* For the first step, we give you some way to provide values for the missing pieces, ie given just a plugin name, determine the group and version.
* The default policy, baked into the core, knows about the built-in plugins (java, groovy), etc and has a hard-coded mapping to provide the missing pieces (group: 'org.gradle', version: GradleVersion.current())
* Your proposal, above, would be an additional, and optional, policy that fills in the missing bits using a web service.
* For the second step, in a similar way, we give you some way to provide the implementation classpath for a given plugin id.
* The default policy would know about the built-in plugins and map their ids to the plugin classloader and/or the distribution lib/plugins directory.
* The default policy could also have some additional mapping, so that given a plugin id (group, plugin, version), attempt to resolve module (group, plugin, version) using the project.repositories and use the resulting jars as the implementation classpath.
Over time, as we bust up the Gradle distribution, the hard-coded mapping can also start to make use of the web service approach. The distinction here would be that we would guarantee there is no ambiguity for the 'official' plugins, so that given a plugin name and gradle version, you always end up with the exact same plugin implementation. So, it is much safer to use defaults.
We can extend this approach as we start to use more meta-data about a plugin:
* To support auto-apply of a plugin. The idea here is that when I run 'gradle wrapper' or 'gradle eclipse', the associated plugin should get automatically applied, even though it is not declared in the build script. We need some way to map from an unknown task name to a plugin. Again, one option is to use a web service that, given a task name, returns a plugin id. Or a list of matches when ambiguous, or a list of candidates when there's no exact match.
* For error messages. There's lots of cools stuff we could do here. Say we encounter `somePlugin { ... }` where there's no extension with the name `somePlugin`, then we can hit the web service to map from extension id to plugin id, and give you an error message saying something like: `No extension 'somePlugin' available. Did you mean to apply plugin 'some-plugin'?`-
We could also use plugin meta-data in the IDE content assistance, where the web service is one source of plugin meta-data:
* Given `apply plugin: `, auto-complete the set of available plugins. This may use the web service to determine the set of available plugins.
* Given `somePlugin { }`, mark `somePlugin` as an unknown extension and offer to add `apply plugin: 'some-plugin'`. This may use the web service to map from extension `somePlugin` to the set of plugins that provide this extension.
And so on... -
-
Very nice...just remember to add caching for this web service :-)
-
-
-
-
-
@Adam/@Merlynn: I think we should move this discussion to a new thread as it's one specific aspect of better plugin support and this thread is getting long.
-
EMPLOYEE
EMPLOYEE
