Integration-Testing Nontrivial Deployments: A Sane Maven Dependency Resolver


The use case which lead to the birth of the ShrinkWrap project may be summed up: identify deployments for simplified integration testing.

So we built an API which made it easy to group resources together in a single deployable unit:

JavaArchive archive = ShrinkWrap.create(JavaArchive.class, "deployment.jar")
   .addClass(MyEjb.class, MyCdiBean.class);

Combined with the Arquillian Test Platform which deploys objects like the above into Application Servers, Servlet Containers, and IoC Containers, the result is a full integration test which writes and executes like a unit test.  This is the heart of the Arquillian component model.

ShrinkWrap excels at filing byte-based content into a tree structure; that’s all a virtual filesystem does (or should do!).  But for a long while now there’s been a divide which has made testing more complex deployments difficult: you might not always have the content you want to insert on-hand.  Consider the following use cases:

  1. A CDI bean which has library dependencies on a 3rd-party project
  2. An EAR module which is composed of a few JARs, WARs, etc. which are built by other projects.
  3. Module A depends on Module B, and you want to easily test the development versions (ie. unreleased) without changing your test code to match

The problem we have here is that once you start calling out to code that’s not in your project, you need to incorporate  dependencies into your deployments.  And finding these dependencies is a nontrivial problem to solve:

  • Not every developer will be keeping a dependency repository in the same place on her machine
  • Versions change as development progresses
  • If you’re testing as part of your build, and your test relies upon a build output (like a JAR), you’ve got a cyclic dependency

This gap is the number one complaint we get in the Arquillian Community, and I’ve been happy to take the blame from the ShrinkWrap side because I’ve refused to offer up stopgap solutions.  I don’t like software that guesses, and I especially don’t like it when software guesses incorrectly.  Behaviour should be clear, concise, and deterministic.

So what we have really is two different problems: First, getting dependency content into a ShrinkWrap archive.  That’s easy.  The harder part is in resolving the dependencies from some coordinates which make sense to put into a test.

Perhaps the biggest asset made available by the Maven community is its expansive Central repository.  Regardless of the build tool you’re using, this repo makes  libraries available under a coordinate system which acts as a unique address (per repository).

Additionally, the Aether project, used by the Maven backend, is a standalone dependency resolver.  Unfortunately, the Aether API isn’t really built to handle the kinds of slimmer-code aims we espouse in the Arquillian community.

Perhaps no one is aware of the warts in testing enterprise software better than the JBoss Quality Engineering team in Brno, and our own Karel Piwko last year started prototyping an adaptor between ShrinkWrap and the Aether backend.  What he built worked, and over the next several months we got a handle on exactly how users were hoping to pull dependencies into their deployments.  We’re now happy with the API and seeking more input to refine it before it locks as backwards-compatible, so:

Tonight we’d like to officially announce ShrinkWrap Resolvers (SWR), 2.0.0-alpha-2.

org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-depchain:2.0.0-alpha-2:pom

The base has no compile-time dependencies (though you’ll need a bunch at runtime), and is capable of resolving to Files, InputStreams, or a custom type if you’d like to provide your own FormatProcessor.

The syntax follows generally the form:

Resolve [this coordinate set] using [some resolution strategy] formatted as [a type]

Its use looks like:

File junit = Maven.resolver().resolve("junit:junit:4.10")
   .withoutTransitivity().asSingle(File.class);

It’s also easy to pull in transitive dependencies associated with a coordinate:

File[] dependencyAndTransitives = Maven.resolver().resolve("groupId:artifactId:version")
   .withTransitivity().as(File.class);

Often we need to run resolution using alternate repositories or project-specific configuration.  In that case:

Maven.configureResolver().fromClassloaderResource("settings.xml")
   .resolve("groupId:artifactId:version").withoutTransitivity().as(File.class);

When we need to test the current version of the software we’re building, we might want to take advantage of all the dependencyManagement settings in our POM such that we don’t have to manually specify the version.  In that case:

Maven.resolver().loadPomFromFile("pom.xml").resolve("groupId:artifactId")
   .withoutTransitivity().as(File.class);

And perhaps we just want to get everything the current POM has defined as a dependency, with correct versions:

Maven.resolver().loadPomFromFile("pom.xml").importRuntimeDependencies()
   .as(File.class)

We’re offering a separate “ShrinkWrap Archive” extension to SWR, enabling direct resolution to an archive:

JavaArchive jar = ShrinkWrapMaven.resolver().resolve("groupId:artifactId:version")
   .withoutTransitivity().asSingle(JavaArchive.class);

SWR also supports offline mode, and integration with a Plugin enables the developer to skip explicitly setting the settings.xml or pom.xml file.

Over the next few alpha-level releases we’ll work on validating that the API makes sense, building up documentation, and porting in more essential features.  As always, we’d love you to get involved in development or offer us your experience through feature requests and bug reports.

S,

ALR

Advertisements

18 comments

  1. Reblogged this on mpashworth and commented:
    Excellent blog about resolving project dependencies into a Shrinkwrap archive for Arquillian. Now I just hope Pax Exam takes note for OSGi testing.

  2. Are there any explicit directions on moving from using EffectivePomMavenDependencyResolver class, ExclusionFilter, etc. ? I’m having some trouble with the class “Maven” resolving to “org.apache.maven.Maven”.

  3. @John: Documentation on the new API will be forthcoming when we lock it as backwards-compatible in Beta. And it looks like beyond that, you’d like a migration guide. Thanks for the suggestion; we’ll be sure to keep that in mind. Though honestly, the migration part might take a backseat priority-wise unless we’ve got some community help in that department (hint hint). 🙂

    “Maven” resolving to “org.jboss.shrinkwrap.resolvers.api.maven” should be as simple as fixing up your IDE’s compile classpath and choosing the right class to import.

    1. Hint received and understood. Thanks for the reply.

  4. I’ve finally had a chance to convert my code to use Alpha2.

    The new API is nice when a simple fluent style is practical, but even harder to use than Alpha1 when you need to do something more complicated.

    In Alpha1 I could write:

    private final EffectivePomMavenDependencyResolver resolver =
    DependencyResolvers.use(MavenDependencyResolver.class)
    .loadEffectivePom(“pom.xml”);

    private final MavenDependencyBuilder depBuilder = resolver.artifacts();

    then add artifacts with `depBuilder.artifacts(“blah:blah”)` to accumulate them.

    In Alpha2 this no longer seems to be possible. It looks like I’ll have to accumulate a list of artifact co-ordinates then pass them to one big resolve() call, or do individual transitive resolutions and build a File[] array. If I do multiple resolutions I’m likely to land up with poor transitive resolution and conflicts, so I’ll probably need to accumulate the artifacts.

    Since `resolve(…)` doesn’t take a `Collection`, only an array, I have to accumulate in a set or list then pass an array view of it to resolve(), eg given a Set artifacts:

    archive.addAsLibraries(resolver.resolve(artifacts.toArray(new String[artifacts.size()])).withTransitivity().as(File.class));

    I don’t think that’s a big problem, but it does go to show that sometimes the push for fluent chaining makes APIs harder to use in the name of superficial ease of use. (I’m particularly looking at .up() in ShrinkWrap Descriptors, here). Fluent is in this regard the new XML – it’s the hammer, and everything becomes a nail.

    More importantly, after moving to alpha3 dependencies that used to resolve now fail to. For example, resolving “commons-collections:commons-collections” worked just fine with alpha1, but with the same pom.xml (except for moving to SW 2.0.0-alpha2) resolution now fails with:

    Caused by: org.jboss.shrinkwrap.resolver.api.ResolutionException: Unable to get version for dependency specified by commons-collections:commons-collections:, it was either null or empty.

    despite the fact that commons-collections is present as a dependency in my pom.xml, when run as:

    Maven.configureResolver().fromFile(“pom.xml”).resolve(“commons-collections:commons-collections”).withTransitivity().as(File.class);

    1. @Craig:

      Thanks for this valuable input; I’d like to address each issue separately. While we have an open window in the 2.0 Alpha series we’re able to change the API to fit all desired use cases. Some have come from our old testsuite (we kept most feature parity), some from the Forge community, some from our own use, etc. If we can identify some other clear use cases from your post here, we’ll accommodate them before the API locks.

      I’d like to be clear that fluency is NOT a design goal; clarity and concision are. In cases where it makes sense to apply fluency, we will. In others (for instance MavenDependency is hierarchical), we won’t. You note ShrinkWrap Descriptors’ “up()” method, our way of handling the hierarchical nature of the data we’re modeling in an object format, and that’s been debated on both sides as well.

      Your issue with “commons-collections” is either user error or a bug, so we’ll iron that out either way.

      Would you mind if I copied this over to our public development forums?

      S,
      ALR

      1. Feel free. I’ll file a JIRA issue for the resolution problem get time to boil it down to a simple test case, too.

  5. * Feature gap issue discussion: https://community.jboss.org/thread/205529
    * Enhancement Request to accept Collections in “resolve” methods: https://issues.jboss.org/browse/SHRINKRES-66
    * Once we can see a test case for the “commons-collections” thing, we’ll fix that, too. Open a discussion on the development forums if you need help figuring where to put your stuff in our test suite, or simply file a JIRA with your standalone case and I’ll port it in.

  6. This api reads naturally well (and reading is more important than writing): nice work 🙂

    The “asSingle(JavaArchive.class)” doesn’t make sense if withTransitivity() is used:
    ShrinkWrapMaven.resolver().resolve(“groupId:artifactId:version”)
    .withTransitivity().asSingle(JavaArchive.class); // BAD
    That will resolve to multiple jars, and those jars shouldn’t be changed: they should certainly not be merged together in an uber jar (because which META-INF manifest.mf and beans.xml file would win?).

    Is there something like asSingle(JavaArchiveCollection.class) to conveniantly add all such jars to a shrinkwrap deployment?

    1. We’d actually considered this case. 🙂

      It *is* possible to resolve something with transitivity and retrieve a single result; if the dependency does not actually have transitives. I’m not sure *why* someone would do this, but it’s surely possible. We make available “asSingle” as a convenience mechanism so that the user doesn’t need to ensure only one item is returned in an array, and get index 0…much like the JPA API works with “getSingleResult()”.

      “asSingle” does *not* merge anything into an uber JAR; it simply ensures that there’s only one result, and returns it. Else a RuntimeException is raised.

      I’m not sure what you’re looking for in your last comment regarding JavaArchiveCollection?

      1. A jar doesn’t support nesting other jars. So I guess what I am looking for is asSingel(WebArchive.class) and have all the jars in there – but that wouldn’t be really handy. I wonder how we can do something like this:

        d = Maven.resolver().loadPomFromFile(“pom.xml”).importRuntimeDependencies(); // This is my normal war
        // add some jars just for the arquillian tests:
        d.addAll(Maven.resolver(.resolve(“org.my:log-to-logging-server:1.0.0”).withTransitivity());
        d.addAll(Maven.resolver(.resolve(“org.my:poll-jboss-as-memory-usage-and-log-it:1.0.0”).withTransitivity());
        d.as(WebArchive.class); // build the war to deploy

        +1 for failing fast with the RuntimeException if asSingle isn’t single.

      2. How is “d” your normal WAR? Your normal WAR would be one artifact, not all runtime dependencies specified by a POM.

        Looks like you’re really just looking to add a bunch of dependencies into your WAR’s WEB-INF/lib? Open a discussion in the forums if more advanced than that, and we’ll work it out.

  7. I am having some problems trying to get all dependencies through getRuntimeDependencies(). I’m trying to get the dependencies from the pom, but the scope of some of them is . How could I add this dependencies with the same approach?

    1. Sorry, I’m not parsing this correctly. 🙂 The scope of some of them is ___ ? Perhaps you need “getRuntimeAndTestDependencies()” (which will bring in all scopes)?

  8. Hi Andrew,

    Thanks for the awesome resolver addition to Shrinkwrap. I am trying to use the resolver library behind a firewall and although I have Nexus proxy and configured the HTTP proxy in the global settings.xml, I still cannot get resolver to use these and the resource it is looking for is in the local maven repo on the hard drive.

    So how do I setup the project to make use of this Maven configuration?

    ——————————————————-
    T E S T S
    ——————————————————-
    Running za.co.testing.arquillian.RegistrationTest
    04 Oct 2012 11:56:45 AM org.jboss.shrinkwrap.resolver.impl.maven.logging.LogTransferListener transferFailed
    WARNING: Failed downloading org/jboss/arquillian/arquillian-bom/1.0.2.Final/arquillian-bom-1.0.2.Final.pom from http://repo1.maven.org/maven2/. Reason
    :
    org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact org.jboss.arquillian:arquillian-bom:pom:1.0.2.Final from/to centra
    l (http://repo1.maven.org/maven2): Error transferring file: Tried all: ‘1’ addresses, but could not connect over HTTP to server: ‘repo1.maven.org’, po
    rt: ’80’ from http://repo1.maven.org/maven2/org/jboss/arquillian/arquillian-bom/1.0.2.Final/arquillian-bom-1.0.2.Final.pom
    Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.211 sec <<< FAILURE!

  9. Any suggestions on the best way to integrate these into the existing boms? Having trouble overriding the existing bom settings for versions.

  10. Shashank · · Reply

    Hello Everyone, I am a newbie to shrinkwrap.
    In my project i am using Arquillian as my integration testing framework and I am trying to resolve a blocking issue.
    I am trying to make a war file using the following code

    MavenDependencyResolver resolver = DependencyResolvers.use(
    MavenDependencyResolver.class).loadMetadataFromPom(“pom.xml”)
    .artifacts(“org.jboss.spec:jboss-javaee:6.0.1.0.0.Final”,…);

    File[] ExtLibs =resolver.resolveAsFiles();
    return ShrinkWrap.create(WebArchive.class, “test.war”). addClasses(LoginModule.class,Realm.class,IAccount.class).addAsLibraries(ExtLibs).addAsManifestResource(EmptyAsset.INSTANCE, “beans.xml”);

    When i run this test case i am getting the following error

    java.lang.RuntimeException: Could not invoke deployment method: public static org.jboss.shrinkwrap.api.spec.WebArchive com.auth.ControllerLoginModuleTest.createDeployment()
    at org.jboss.arquillian.container.test.impl.client.deployment.AnnotationDeploymentScenarioGenerator.invoke(AnnotationDeploymentScenarioGenerator.java:160)

    I don’t iunderstand what i am doing wrong. Any light on the above issue is greatly appreciated.
    Thanks

    1. Hiya! Is this still relevant for you? If so, have a look at the user guide for ShrinkWrap Resolvers 2.0.0 here: https://github.com/shrinkwrap/resolver/blob/master/README.asciidoc

      From this error, looks like we’re missing the cause of the RuntimeException; would need some more context. If this persists, open a discussion on our user forums!

      S,
      ALR

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: