[issue13959] Re-implement parts of imp in pure Python

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
58 messages Options
123
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor

New submission from Brett Cannon <[hidden email]>:

A bunch of code in Python/import.c exists purely for the imp module, but a large chunk of it does not need to be implemented in C but instead can be implemented in Python code.

Once importlib is bootstrapped in then an attempt to scale back the C code should be done by re-implementing parts of imp in pure Python (either in some _imp module or directly in _importlib itself).

----------
components: Library (Lib)
messages: 152800
nosy: brett.cannon
priority: normal
severity: normal
status: open
title: Re-implement parts of imp in pure Python
versions: Python 3.3

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor

Changes by Brett Cannon <[hidden email]>:


----------
dependencies: +Replace __import__ w/ importlib.__import__

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Changes by Éric Araujo <[hidden email]>:


----------
nosy: +eric.araujo

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Changes by Arfrever Frehtes Taifersar Arahesis <[hidden email]>:


----------
nosy: +Arfrever

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Changes by Meador Inge <[hidden email]>:


----------
nosy: +meador.inge

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Changes by Brett Cannon <[hidden email]>:


----------
nosy: +eric.snow

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Brett Cannon <[hidden email]> added the comment:

Attached is a start for re-implementing imp. Still need code for cache_from_source, source_from_cache, find_module, load_module, reload, load_compiled, load_source, and load_package.

----------
keywords: +patch
Added file: http://bugs.python.org/file24846/imp.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Eric Snow <[hidden email]> added the comment:

Here's an incomplete (though essentially test-passing) patch that flips more of imp into importlib.  It builds on Brett's patch.  The gist is that imp.py is a minimal wrapper around importlib, which uses _imp directly.

Ultimately the things implemented in Lib/importlib/_bootstrap.py should be yanked out of Python/import.c (and Python/dynload_*.c).  I expect that this will wait for the bootstrap merge (#2377) to land.

----------
Added file: http://bugs.python.org/file24979/imp.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Eric Snow <[hidden email]> added the comment:

Here's a listing of the places where import-ish modules are used in CPython.  Once things all the import stuff currently on the table is wrapped up, I'll probably work on switching things over to using importlib directly.

----------
Added file: http://bugs.python.org/file24980/use_of_imp.txt

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Eric Snow <[hidden email]> added the comment:

Of note for my patch is the addition of SUFFIXES and MODE to the 3 main "file loader" classes.  I did this to reverse the dependence on imp.get_suffixes().

As well, a couple of extra functions are added to Python/importlib/_bootstrap.py for a similar reason, but should also be meaningful externally in their own right.

load_module_from_file(), is one of these, though the current incarnation relies on a mythical new method on loaders that may not pass muster.

(FWIW, I accidently included the use_of_imp.txt file in the patch and don't intend it to get included)

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Éric Araujo <[hidden email]> added the comment:

> Once things all the import stuff currently on the table is wrapped up,
> I'll probably work on switching things over to using importlib directly.

Please leave distutils unchanged; we don’t clean it up or improve it in any way, it’s bugfixes only.  Changing packaging is fine; I’ll just keep using the imp module in the distutils2-python3 backport.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Eric Snow <[hidden email]> added the comment:

Sounds good.  It will be a while before we get there, but at that point I was already planning on getting in touch with the maintainers of relevant modules before messing with them.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Eric Snow <[hidden email]> added the comment:

(Éric, I'd reply in the review if reitveld recognized my current username -- see http://psf.upfronthosting.co.za/roundup/meta/issue402.  FYI, I also did not get an email for your review, which is likely related to that issue.)

Regarding __all__, I didn't realize that about being a tuple.  I'll fix it.

As to the "dm" in the SOABI value, that is definitely something we'll have to factor in.  There are actually a few such places in the current patch that rely on problematic data.   It's either hidden in C or not available during bootstrapping.  That's something we'll have to iron out before the solution for this issue gets committed.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Antoine Pitrou <[hidden email]> added the comment:

I think this should be a blocker for 3.3.

----------
nosy: +pitrou
priority: normal -> release blocker

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Brett Cannon <[hidden email]> added the comment:

Just because I was thinking about it, I wonder if necessarily all the frozen stuff really needs to stay in import.c. I mean a frozen module is really just an entry in an array of structs that has a name of an char*[]. I don't see why one couldn't simply have a get_frozen_bytes() method to convert that char*[] into a bytes object and use that to construct a module in pure Python code.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Roundup Robot <[hidden email]> added the comment:

New changeset d777f854a66e by Brett Cannon in branch 'default':
Issue #13959: Rename imp to _imp and add Lib/imp.py and begin
http://hg.python.org/cpython/rev/d777f854a66e

----------
nosy: +python-dev

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Brett Cannon <[hidden email]> added the comment:

OK, so I have started to check this stuff in, but I think it's best to do it piecemeal. Going forward I would like to commit in units of functions being replaced, and prioritize stuff that cuts out C code (e.g. the load_*() methods, find_module(), etc.). That way it's clear that progress is being made. Obviously the best way to tell if code is hanging on just because of imp is to comment out the interface code and see what your compiler complains about in terms of dead code (or at least clang is good at this).

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Brett Cannon <[hidden email]> added the comment:

It looks like in order to get a clear sign of what it will take to remove various parts of import.c, imp.load_module() needs to go along with imp.load_package() (since they call each other in the C code). You also have to take care of imp.reload(), but I am simplifying the C code greatly and will take care of referencing other code in the module.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Roundup Robot <[hidden email]> added the comment:

New changeset 4dce3afc392c by Brett Cannon in branch 'default':
Issue #13959: Simplify imp.reload() by relying on a module's
http://hg.python.org/cpython/rev/4dce3afc392c

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue13959] Re-implement parts of imp in pure Python

STINNER Victor
In reply to this post by STINNER Victor

Brett Cannon <[hidden email]> added the comment:

I am seeing how this is going to go down. the load_dynamic, load_source, etc. family of functions are simply dispatched to by load_module(). So to keep some semblance of backwards-compatibility, each of those modules need to be implemented and then have load_module() simply dispatch to them based on the "type" of module it is.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

123
Loading...