Unshaded uberjars

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Unshaded uberjars

Curtis Rueden
Hi everyone,

First of all, thanks to the Jython community for all the hard work!

My group uses Jython in the ImageJ project [1]. In recent months we have had an issue [2] with shipping both Jython (2.5.3) and JRuby (1.7.12), since they rely on incompatible versions of JFFI. Only one of Jython or JRuby would work, depending on ordering of libraries on the classpath.

In short: JFFI broke binary compatibility so regardless of the classpath order, there is no "correct" JFFI version for both projects.

Both org.python:jython:2.5.3 and org.python:jython-standalone:2.5.3 are unshaded uberjars, meaning they bundle all their dependencies without renaming the packages. This makes it more difficult to include alternate versions of said dependencies, especially if there is a second uberjar which includes a differing version.

To work around this problem, I created a "jython-shaded" artifact for 2.5.3 with shaded dependencies, and released it to OSS Sonatype (and hence Maven Central) under the org.scijava groupId. Details at: https://github.com/scijava/jython-shaded

Using jython-shaded instead of jython-standalone allows both Jython and JRuby scripts to coexist peacefully in the same JVM, without resorting to a ClassLoader-based solution like OSGi.

I'm happy to contribute the project upstream if the maintainers are interested; otherwise, it can continue to live happily in org.scijava.

Cheers,
Curtis



------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Fwd: Unshaded uberjars

Jim Baker-2
+list

---------- Forwarded message ----------
From: Jim Baker <[hidden email]>
Date: Thu, Sep 11, 2014 at 3:16 PM
Subject: Re: [Jython-users] Unshaded uberjars
To: Curtis Rueden <[hidden email]>


Curtis,

In general, the Maven shade plugin is more or less equivalent to jarjarlinks, which is what Jython currently uses in an Ant task.

So we have renaming rules like the following:

    <target name="jar-complete" depends="jar">
        <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask" classpath="extlibs/jarjar-1.4.jar"/>
        <jarjar destfile="${dist.dir}/${jython.deploy.jar}">
        ...
   <zipfileset src="extlibs/bcpkix-jdk15on-150.jar" excludes="META-INF/*.SF"/>
   <rule pattern="org.bouncycastle.**" result="org.python.bouncycastle.@1"/>

which ensures that bouncycastle is put into the org.python namespace, to avoid jar incompatibility issues.

The approach Maven takes is much better, because this renaming is often overkill - it really should be done as late as possible when the final assembly is being built, so as to avoid unnecessary copies and uber jar bloat. There's an outstanding bug to move to a Maven-based build, while using Ant for such core pieces as building out the grammar or processing annotations for exposed classes. See http://bugs.jython.org/issue2182

Specifically, we should do renaming for JFFI as you suggested. For the 2.5.4 release, we should add this to the above Ant task; for 2.7, we probably will continue using Ant for renaming for 2.7.0, but I hope to see Maven usage by 2.7.1, at which point we can adopt the solution you have done.

- Jim

On Thu, Sep 11, 2014 at 11:34 AM, Curtis Rueden <[hidden email]> wrote:
Hi everyone,

First of all, thanks to the Jython community for all the hard work!

My group uses Jython in the ImageJ project [1]. In recent months we have had an issue [2] with shipping both Jython (2.5.3) and JRuby (1.7.12), since they rely on incompatible versions of JFFI. Only one of Jython or JRuby would work, depending on ordering of libraries on the classpath.

In short: JFFI broke binary compatibility so regardless of the classpath order, there is no "correct" JFFI version for both projects.

Both org.python:jython:2.5.3 and org.python:jython-standalone:2.5.3 are unshaded uberjars, meaning they bundle all their dependencies without renaming the packages. This makes it more difficult to include alternate versions of said dependencies, especially if there is a second uberjar which includes a differing version.

To work around this problem, I created a "jython-shaded" artifact for 2.5.3 with shaded dependencies, and released it to OSS Sonatype (and hence Maven Central) under the org.scijava groupId. Details at: https://github.com/scijava/jython-shaded

Using jython-shaded instead of jython-standalone allows both Jython and JRuby scripts to coexist peacefully in the same JVM, without resorting to a ClassLoader-based solution like OSGi.

I'm happy to contribute the project upstream if the maintainers are interested; otherwise, it can continue to live happily in org.scijava.

Cheers,
Curtis



------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users




--



--

------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Unshaded uberjars

Curtis Rueden
Hi Jim,

Thanks for the informative reply. All sounds good. If there is anything I can do to help, just let me know.

Regards,
Curtis

On Thu, Sep 11, 2014 at 5:17 PM, Jim Baker <[hidden email]> wrote:
+list

---------- Forwarded message ----------
From: Jim Baker <[hidden email]>
Date: Thu, Sep 11, 2014 at 3:16 PM
Subject: Re: [Jython-users] Unshaded uberjars
To: Curtis Rueden <[hidden email]>


Curtis,

In general, the Maven shade plugin is more or less equivalent to jarjarlinks, which is what Jython currently uses in an Ant task.

So we have renaming rules like the following:

    <target name="jar-complete" depends="jar">
        <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask" classpath="extlibs/jarjar-1.4.jar"/>
        <jarjar destfile="${dist.dir}/${jython.deploy.jar}">
        ...
   <zipfileset src="extlibs/bcpkix-jdk15on-150.jar" excludes="META-INF/*.SF"/>
   <rule pattern="org.bouncycastle.**" result="org.python.bouncycastle.@1"/>

which ensures that bouncycastle is put into the org.python namespace, to avoid jar incompatibility issues.

The approach Maven takes is much better, because this renaming is often overkill - it really should be done as late as possible when the final assembly is being built, so as to avoid unnecessary copies and uber jar bloat. There's an outstanding bug to move to a Maven-based build, while using Ant for such core pieces as building out the grammar or processing annotations for exposed classes. See http://bugs.jython.org/issue2182

Specifically, we should do renaming for JFFI as you suggested. For the 2.5.4 release, we should add this to the above Ant task; for 2.7, we probably will continue using Ant for renaming for 2.7.0, but I hope to see Maven usage by 2.7.1, at which point we can adopt the solution you have done.

- Jim

On Thu, Sep 11, 2014 at 11:34 AM, Curtis Rueden <[hidden email]> wrote:
Hi everyone,

First of all, thanks to the Jython community for all the hard work!

My group uses Jython in the ImageJ project [1]. In recent months we have had an issue [2] with shipping both Jython (2.5.3) and JRuby (1.7.12), since they rely on incompatible versions of JFFI. Only one of Jython or JRuby would work, depending on ordering of libraries on the classpath.

In short: JFFI broke binary compatibility so regardless of the classpath order, there is no "correct" JFFI version for both projects.

Both org.python:jython:2.5.3 and org.python:jython-standalone:2.5.3 are unshaded uberjars, meaning they bundle all their dependencies without renaming the packages. This makes it more difficult to include alternate versions of said dependencies, especially if there is a second uberjar which includes a differing version.

To work around this problem, I created a "jython-shaded" artifact for 2.5.3 with shaded dependencies, and released it to OSS Sonatype (and hence Maven Central) under the org.scijava groupId. Details at: https://github.com/scijava/jython-shaded

Using jython-shaded instead of jython-standalone allows both Jython and JRuby scripts to coexist peacefully in the same JVM, without resorting to a ClassLoader-based solution like OSGi.

I'm happy to contribute the project upstream if the maintainers are interested; otherwise, it can continue to live happily in org.scijava.

Cheers,
Curtis



------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users




--



--

------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk

_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users



------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users