What's the best way to dispose a PythonInterpreter?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

What's the best way to dispose a PythonInterpreter?

Steve Yegge
I create PythonInterpreter instances on demand when admins log into my system.  When they leave I'd like to clean up their interpreter instance.

Do I just call cleanup() and then remove any references to it?  What exactly does PySystemState.callExitFunc do?  Is it just a customization hook that I can safely assume is null if I didn't set it?

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: What's the best way to dispose a PythonInterpreter?

Indra Talip
Steve,
Sadly I don't think that is sufficient. The "sys" module (aka PySystemState) exists as a ThreadLocal that the jython modules written in java access through Py.getSystemState(). BTW this turns out to be a real pain in the neck when you use a thread pool to execute code for different PythonInterpreters (it requires me to track the PySystemState that should be used for each PythonInterpreter and call Py.setSystemState() with the correct PySystemState before invoking any of my python code).

I don't think there is a way via Py.java to clear the thread state (happy to be corrected). For my use case I ended up not bothering to throw away the PythonInterpreter and attempt to (with some shenanigans related to import hooks) delete and reload modules that have changed. I also however would prefer that I could just call cleanup() and remove all references and that would be it.

Even the (possibly overkill) approach of loading each PythonInterpreter into it's own ClassLoader probably won't work/will leak memory at the moment due to memory leaks related to subclassing Thread (c.f. https://bitbucket.org/jython/jython/pull-request/19/pysystemstateclosershutdowncloser-use and http://bugs.jython.org/issue2127).

Does anyone else have any suggestions?

Cheers
Indra

On 10 May 2014 17:01, Steve Yegge <[hidden email]> wrote:
I create PythonInterpreter instances on demand when admins log into my system.  When they leave I'd like to clean up their interpreter instance.

Do I just call cleanup() and then remove any references to it?  What exactly does PySystemState.callExitFunc do?  Is it just a customization hook that I can safely assume is null if I didn't set it?

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users




--
Indra Talip

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: What's the best way to dispose a PythonInterpreter?

Jim Baker-2
We are going to fix these resource leak issues to the extent possible for the 2.7.0 release. Problems we have seen now or in the past are as follows:

Thread locals. In the case of PythonInterpreter, we only added them with setLocals, as used by JSR 223, we never removed. Obvious opportunity for a memory leak! See http://bugs.jython.org/issue2026, which is currently in progress, include support for try-with-resources as now available in Java 7. So PythonInterpreter#close (which will be a new synonym from Closeable/Autocloseable for the cleanup method) means cleaning all aspects of this PythonInterpreter. So Indra, you will get what you want: "prefer that I could just call cleanup() and remove all references and that would be it"

There's a more subtle problem associated with thread locals in that they can also leak ClassLoaders. ThreadState is a good example of where this has been a problem in the past (http://bugs.jython.org/issue1327). We should also apply an additional step of indirection, as described in http://blog.crazybob.org/2006/07/hard-core-java-threadlocal.html

Caches. I believe we are mostly pass this as a problem, thanks to using Google Guava collections and especially its support for caches built using weakrefs.

Mutable static fields holding hard references. PySystemState is problematic here. We partially addressed this with a shadowing mechanism in http://hg.python.org/jython/rev/62d5016fe64e, but we need to do a clean sweep and remove the remainder in favor of making these instance fields, as well as make PySystemState#shadow a deprecated no-op. Note this will be a breaking API change, but something we must do for 2.7.

Subclassed threads. We see this in http://bugs.jython.org/issue2127 (under review) and recently fixed in http://hg.python.org/jython/rev/55f7eaddc954 Use Runnable instead.

None of this is too difficult to implement, assuming we can break APIs, but it can be hard to duplicate specific leak scenarios. However, I think we know best practices here and we should just start applying them.

- Jim


On Sat, May 10, 2014 at 1:58 AM, Indra Talip <[hidden email]> wrote:
Steve,
Sadly I don't think that is sufficient. The "sys" module (aka PySystemState) exists as a ThreadLocal that the jython modules written in java access through Py.getSystemState(). BTW this turns out to be a real pain in the neck when you use a thread pool to execute code for different PythonInterpreters (it requires me to track the PySystemState that should be used for each PythonInterpreter and call Py.setSystemState() with the correct PySystemState before invoking any of my python code).

I don't think there is a way via Py.java to clear the thread state (happy to be corrected). For my use case I ended up not bothering to throw away the PythonInterpreter and attempt to (with some shenanigans related to import hooks) delete and reload modules that have changed. I also however would prefer that I could just call cleanup() and remove all references and that would be it.

Even the (possibly overkill) approach of loading each PythonInterpreter into it's own ClassLoader probably won't work/will leak memory at the moment due to memory leaks related to subclassing Thread (c.f. https://bitbucket.org/jython/jython/pull-request/19/pysystemstateclosershutdowncloser-use and http://bugs.jython.org/issue2127).

Does anyone else have any suggestions?

Cheers
Indra

On 10 May 2014 17:01, Steve Yegge <[hidden email]> wrote:
I create PythonInterpreter instances on demand when admins log into my system.  When they leave I'd like to clean up their interpreter instance.

Do I just call cleanup() and then remove any references to it?  What exactly does PySystemState.callExitFunc do?  Is it just a customization hook that I can safely assume is null if I didn't set it?

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users




--
Indra Talip

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users



------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Fwd: What's the best way to dispose a PythonInterpreter?

Jim Baker-2
reply to list

---------- Forwarded message ----------
From: Jim Baker <[hidden email]>
Date: Fri, Jun 20, 2014 at 11:23 AM
Subject: Re: [Jython-users] What's the best way to dispose a PythonInterpreter?
To: Steve Yegge <[hidden email]>
Cc: Indra Talip <[hidden email]>


Just to be clear, by broken APIs I mean having to use getters/setters in the Java API and not directly access via Java fields, at least with respect to static fields.

Example: use PySystemState#setDefaultEncoding, instead of manipulating defaultEncoding, because such fields are going to become private and nonstatic. This is a good example where we were perhaps too conservative in our API changes and implemented a complex "shadowing" mechanism to support both old and new style usage together.

So nothing controversial in this refactoring, just good coding practice, and the sort of thing that users of the Java API can adjust to in a few minutes of work, or possibly just recompile. Users from Python will of course not even see a difference.

- Jim


On Fri, Jun 20, 2014 at 1:11 AM, Steve Yegge <[hidden email]> wrote:
Thanks for the update, Jim.  I don't mind broken APIs to get this functionality.  Looking forward to it.


On Thu, Jun 19, 2014 at 4:32 PM, Jim Baker <[hidden email]> wrote:
We are going to fix these resource leak issues to the extent possible for the 2.7.0 release. Problems we have seen now or in the past are as follows:

Thread locals. In the case of PythonInterpreter, we only added them with setLocals, as used by JSR 223, we never removed. Obvious opportunity for a memory leak! See http://bugs.jython.org/issue2026, which is currently in progress, include support for try-with-resources as now available in Java 7. So PythonInterpreter#close (which will be a new synonym from Closeable/Autocloseable for the cleanup method) means cleaning all aspects of this PythonInterpreter. So Indra, you will get what you want: "prefer that I could just call cleanup() and remove all references and that would be it"

There's a more subtle problem associated with thread locals in that they can also leak ClassLoaders. ThreadState is a good example of where this has been a problem in the past (http://bugs.jython.org/issue1327). We should also apply an additional step of indirection, as described in http://blog.crazybob.org/2006/07/hard-core-java-threadlocal.html

Caches. I believe we are mostly pass this as a problem, thanks to using Google Guava collections and especially its support for caches built using weakrefs.

Mutable static fields holding hard references. PySystemState is problematic here. We partially addressed this with a shadowing mechanism in http://hg.python.org/jython/rev/62d5016fe64e, but we need to do a clean sweep and remove the remainder in favor of making these instance fields, as well as make PySystemState#shadow a deprecated no-op. Note this will be a breaking API change, but something we must do for 2.7.

Subclassed threads. We see this in http://bugs.jython.org/issue2127 (under review) and recently fixed in http://hg.python.org/jython/rev/55f7eaddc954 Use Runnable instead.

None of this is too difficult to implement, assuming we can break APIs, but it can be hard to duplicate specific leak scenarios. However, I think we know best practices here and we should just start applying them.

- Jim


On Sat, May 10, 2014 at 1:58 AM, Indra Talip <[hidden email]> wrote:
Steve,
Sadly I don't think that is sufficient. The "sys" module (aka PySystemState) exists as a ThreadLocal that the jython modules written in java access through Py.getSystemState(). BTW this turns out to be a real pain in the neck when you use a thread pool to execute code for different PythonInterpreters (it requires me to track the PySystemState that should be used for each PythonInterpreter and call Py.setSystemState() with the correct PySystemState before invoking any of my python code).

I don't think there is a way via Py.java to clear the thread state (happy to be corrected). For my use case I ended up not bothering to throw away the PythonInterpreter and attempt to (with some shenanigans related to import hooks) delete and reload modules that have changed. I also however would prefer that I could just call cleanup() and remove all references and that would be it.

Even the (possibly overkill) approach of loading each PythonInterpreter into it's own ClassLoader probably won't work/will leak memory at the moment due to memory leaks related to subclassing Thread (c.f. https://bitbucket.org/jython/jython/pull-request/19/pysystemstateclosershutdowncloser-use and http://bugs.jython.org/issue2127).

Does anyone else have any suggestions?

Cheers
Indra

On 10 May 2014 17:01, Steve Yegge <[hidden email]> wrote:
I create PythonInterpreter instances on demand when admins log into my system.  When they leave I'd like to clean up their interpreter instance.

Do I just call cleanup() and then remove any references to it?  What exactly does PySystemState.callExitFunc do?  Is it just a customization hook that I can safely assume is null if I didn't set it?

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users




--
Indra Talip

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users






------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users