SysPackageManager to introspect a .war file?

[Marcin Cieslak-3]

Hello,

I am trying to add Jython console to Gerrit (http://code.google.com/p/gerrit/).

I got the basics working - now I am able to launch Jython's InteractiveConsole
from within an application, started by running :

% java -jar gerrit-2.3.war daemon <..args...>

(The WAR file can be downloaded from
 http://code.google.com/p/gerrit/downloads/detail?name=gerrit-2.3.war)

A standalone jython.jar gets loaded from the application "lib" directory
automatically and my code is setting up Jython this way:

    final String CACHEDIR = "python.cachedir";
    final String CACHEDIR_SKIP = "python.cachedir.skip";
    Properties post = new Properties();

    post.setProperty(CACHEDIR, "/tmp/cachedir");
    post.setProperty(CACHEDIR_SKIP, "0");

It then calls (via reflection API):

    PySystemState.initialize(null, post)
    console = new InteractiveConsole();
    console.invoke("Some banner", null);

Jython (as expected) is finding all classes on the current Java
classloader (so "import com.google.gerrit.launcher.GerritLauncher" works)
but it does not know all the packages in advance, because SysPackageManager
does not introspect a WAR file.

Is there any way to make SysPackageManager to introspect a WAR file
and load all JarFiles from "WEB-INF/lib" and classes from "WEB-INF/classes"?

Can I spoonfeed the SysPackageManager with packages some other way than
just importing all packages (there are *lots* of them)?

Below please find a sample interactive session plus dump of the registry.

//Marcin

$ java -jar gerrit\-war/target/gerrit\-2.3\-SNAPSHOT.war daemon -d ../test\-site-pg -s
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/resources.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/rt.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/jsse.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/jce.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/charsets.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/ext/sunpkcs11.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/ext/dnsns.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/ext/localedata.jar'
*sys-package-mgr*: processing new jar, '/usr/local/openjdk6/jre/lib/ext/sunjce_provider.jar'
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[OpenJDK 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0 running for Gerrit 2.3-rc0-160-g7a9bb6a
>>> import com.google
>>> dir(com.google)
['__name__']
>>> import com.google.gerrit.launcher.GerritLauncher
>>> dir(com.google.gerrit.launcher.GerritLauncher)
['NOT_ARCHIVED', '__class__', '__copy__', '__deepcopy__', '__delattr__', '__doc__', '__eq__', '__getattribute__', '__hash__', '__init__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__unicode__', 'class', 'createTempFile', 'equals', 'getClass', 'getDistributionArchive', 'hashCode', 'main', 'notify', 'notifyAll', 'toString', 'wait']
>>> dir(com.google)
['__name__', 'gerrit']
>>> import sys
>>> for k in sys.registry.keys():
...      print "%s=%s" % (k, sys.registry[k])
...
java.runtime.name=OpenJDK Runtime Environment
python.cachedir=/tmp/cachedir
sun.boot.library.path=/usr/local/openjdk6/jre/lib/amd64
java.vm.version=20.0-b12
java.vm.vendor=Sun Microsystems Inc.
java.vendor.url=http://java.sun.com/
path.separator=:
java.vm.name=OpenJDK 64-Bit Server VM
file.encoding.pkg=sun.io
sun.java.launcher=SUN_STANDARD
user.country=PL
sun.os.patch.level=unknown
java.vm.specification.name=Java Virtual Machine Specification
user.dir=/home/saper/sw/gerrit
java.runtime.version=1.6.0-b24
java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
java.endorsed.dirs=/usr/local/openjdk6/jre/lib/endorsed
os.arch=amd64
java.io.tmpdir=/var/tmp/
line.separator=

java.vm.specification.vendor=Sun Microsystems Inc.
os.name=FreeBSD
sun.jnu.encoding=UTF-8
java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
java.specification.name=Java Platform API Specification
java.class.version=50.0
sun.management.compiler=HotSpot 64-Bit Tiered Compilers
os.version=8.2-STABLE
user.home=/home/saper
user.timezone=GMT
java.awt.printerjob=sun.print.PSPrinterJob
file.encoding=UTF-8
java.specification.version=1.6
java.class.path=gerrit-war/target/gerrit-2.3-SNAPSHOT.war
user.name=saper
python.console.encoding=UTF-8
java.vm.specification.version=1.0
sun.java.command=gerrit-war/target/gerrit-2.3-SNAPSHOT.war daemon -d ../test-site-pg -s
java.home=/usr/local/openjdk6/jre
sun.arch.data.model=64
user.language=pl
java.specification.vendor=Sun Microsystems Inc.
java.vm.info=mixed mode
java.version=1.6.0
java.ext.dirs=/usr/local/openjdk6/jre/lib/ext:/usr/java/packages/lib/ext
sun.boot.class.path=/usr/local/openjdk6/jre/lib/resources.jar:/usr/local/openjdk6/jre/lib/rt.jar:/usr/local/openjdk6/jre/lib/sunrsasign.jar:/usr/local/openjdk6/jre/lib/jsse.jar:/usr/local/openjdk6/jre/lib/jce.jar:/usr/local/openjdk6/jre/lib/charsets.jar:/usr/local/openjdk6/jre/lib/modules/jdk.boot.jar:/usr/local/openjdk6/jre/classes
java.vendor=Sun Microsystems Inc.
net.sf.ehcache.skipUpdateCheck=true
python.cachedir.skip=0
file.separator=/
java.vendor.url.bug=http://java.sun.com/cgi-bin/bugreport.cgi
sun.io.unicode.encoding=UnicodeLittle
sun.cpu.endian=little
sun.cpu.isalist=


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users

[Alan Kennedy-2]

[Marcin]
> I am trying to add Jython console to Gerrit (http://code.google.com/p/gerrit/).

Cool.

> Is there any way to make SysPackageManager to introspect a WAR file
> and load all JarFiles from "WEB-INF/lib" and classes from "WEB-INF/classes"?

Modjy, our WSGI->servlets gateway, supports this. Perhaps you could
take a look at how it works.

Take a look at the definition of the "packages", "classdirs" and
"extdirs" parameters.

http://opensource.xhaus.com/projects/modjy/wiki/ModjyConfiguration

Read also the "Using external java packages" note at the end of that page.

Here is how those parameters are processed

http://hg.python.org/jython/file/99b8d4f16639/Lib/modjy/modjy_impl.py#l77

Basically, those parameters are passed to

sys.add_package
sys.add_classdir
sys.add_extdir

Which are methods of PySystemState.

http://hg.python.org/jython/file/99b8d4f16639/src/org/python/core/PySystemState.java#l1164

You may need to call ServletContext.getRealPath("WEB-INF/classes"),
etc, to get the path to pass to those methods.

Let us know how you get on.

Alan.

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users

[Marcin Cieslak-3]

>> Alan Kennedy <[hidden email]> wrote:
> [Marcin]
>> I am trying to add Jython console to Gerrit (http://code.google.com/p/gerrit/).
>
> Cool.
>
>> Is there any way to make SysPackageManager to introspect a WAR file
>> and load all JarFiles from "WEB-INF/lib" and classes from "WEB-INF/classes"?
>
> Modjy, our WSGI->servlets gateway, supports this. Perhaps you could
> take a look at how it works.
>
> Take a look at the definition of the "packages", "classdirs" and
> "extdirs" parameters.

Thank you. It seems like those have to be explicitly specified; and
I wanted to get them all automatically. I can't use ServletContext
or other servlet features because Gerrit is a standalone Java
application that starts Jetty as one of its sub-components (therefore
not everything runs within the servlet container).

I found out that the bootstrap code of Gerrit is just unpacking
a WAR into the temporary directory and then using java.net.URLClassLoader
to load all WEB-INF/lib classes.

The following trick did it for me:

    StringBuilder classPath = new StringBuilder();
    final ClassLoader cl = getClass().getClassLoader();
    if (cl instanceof java.net.URLClassLoader) {
      URLClassLoader ucl = (URLClassLoader)cl;
      for (URL u : ucl.getURLs()) {
        if ("file".equals(u.getProtocol())) {
          if (classPath.length() > 0) {
            classPath.append(java.io.File.pathSeparatorChar);
          }
          classPath.append(u.getFile());
        }
      }
    }
    post.setProperty(CLASSPATH, classPath.toString());

The above code runs before setting up PySystemState.

If there was a custom classloader it could be converted to support
Jython straight away. Maybe even Jython community should advocate some
standard interface for customer classloaders to supply list of available
classes...

But some way to nicely add "WEB-INF/class", "WEB-INF/lib" and
recursive jar scanning somewhere to the cache manager
like org.python.core.packagecache.addWarToPackages() would be
useful.

Here is my patch to include support for Jython in Gerrit:

  https://gerrit-review.googlesource.com/#/c/34670/

//Marcin



------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users