Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

Jim Baker-2
Eric,

Something like these subinterpreters in CPython are used from Jython's Java API. Like nearly all of Jython* this can be directly imported into using Python code, as seen in tests using this feature:  https://github.com/jythontools/jython/blob/master/Lib/test/test_pythoninterpreter_jy.py

More on the API here: https://github.com/jythontools/jython/blob/master/src/org/python/util/PythonInterpreter.java - note that is not even a core API for Jython, it just happens to be widely used, including by the launcher that wraps this API and calls itself the jython executable. So we can readily refactor if we have something better, because right now it is also problematic with respect to its lifecycle; what is the mapping to threads; and how it interacts with class loaders and other resources, especially during cleanup.

It would be helpful to coordinate this subinterpreter work; or at least to cc jython-dev on such ideas your might have. Recently there have been some rumblings of consensus that it's about time for Jython to really start work on the 3.x implementation, targeting 3.6. But do be aware we are usually at most 2 to 5 developers, working in our spare time. So everything takes much longer than one would hope. I just hope we can finish 3.6 (or whatever) before Python 4.0 arrives :)

*Excluding certain cases on core types where our bytecode rewriting makes it a true challenge!

- Jim


On Thu, May 25, 2017 at 11:03 AM, Eric Snow <[hidden email]> wrote:
On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum <[hidden email]> wrote:
> Hm... Curiously, I've heard a few people at PyCon

I'd love to get in touch with them and discuss the situation.  I've
spoken with Graham Dumpleton on several occasions about
subinterpreters and what needs to be fixed.

> mention they thought subinterpreters were broken

There are a number of related long-standing bugs plus a few that I
created in the last year or two.  I'm motivated to get these resolved
so that the multi-core Python project can take full advantage of
subinterpreters without worry.

As well, there are known limitations to using extension modules in
subinterpreters.  However, only extension modules that rely on process
globals (rather than leveraging PEP 384, etc.) are affected, and we
can control for that more carefully using the protocol introduced by
PEP 489.

There isn't anything broken about the concept or design of
subinterpreters in CPython that I'm aware of.

> and not useful (and they share the GIL anyways)

I'll argue that their usefulness has been limited by lack of exposure
in the stdlib. :)  Furthermore, I'm finding them extremely useful as
the vehicle for the multi-core Python project.

> and should be taken out. So we should at least have clarity on which
> direction we want to take...

I'd definitely appreciate a firm commitment that they are not getting
removed as I don't want to spend a ton of time on the project just to
have the effort made irrelevant. :)  Also, I'd be surprised if there
were sufficient merit to removing support for subinterpreters, since
there is very little machinery just for that feature.  Instead, it
leverages components of CPython that are there for other valid
reasons.  So I do not consider subinterpreters to currently add any
significant burden to maintenance or development of the code base.
Regardless, exposing the low-level _subinterpreters module should help
us iron out bugs and API, as Nick pointed out.

-eric
_______________________________________________
Python-ideas mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Jython-dev mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-dev