path joining on Windows and imp.cache_from_source()

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

path joining on Windows and imp.cache_from_source()

Brett Cannon-2
imp.cache_from_source() (and thus also imp.source_from_cache()) has special semantics compared to how os.path.join() works. For instance, if you look at test_imp you will notice it tries to use the same path separator as is the farthest right in the path it is given::

  self.assertEqual(imp.cache_from_source('\\foo\\bar/baz/qux.py', True), '\\foo\\bar\\baz/__pycache__/qux.{}.pyc'.format(self.tag))

But if you do the same basic operation using ntpath, you will notice it simply doesn't care::

  >>> ntpath.join(ntpath.split('a\\b/c/d.py')[0], '__pycache__', 'd.cpython-32.pyc')
  'a\\b/c\\__pycache__\\d.cpython-32.pyc

Basically imp.cache_from_source() goes to a bunch of effort to reuse the farthest right separator when there is an alternative separator *before* and path splitting is done. But if you look at ntpath.join(), it doesn't even attempt that much effort.

Now that we can reuse os.path.join() (directly for source_from_cache(), indirectly through easy algorithmic copying in cache_from_source()) do we want to keep the "special" semantics, or can I change it to match what ntpath would do when there can be more than one path separator on an OS (i.e. not do anything special)?

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: path joining on Windows and imp.cache_from_source()

Glenn Linderman-3
On 4/21/2012 8:53 PM, Brett Cannon wrote:
imp.cache_from_source() (and thus also imp.source_from_cache()) has special semantics compared to how os.path.join() works. For instance, if you look at test_imp you will notice it tries to use the same path separator as is the farthest right in the path it is given::

  self.assertEqual(imp.cache_from_source('\\foo\\bar/baz/qux.py', True), '\\foo\\bar\\baz/__pycache__/qux.{}.pyc'.format(self.tag))

But if you do the same basic operation using ntpath, you will notice it simply doesn't care::

  >>> ntpath.join(ntpath.split('a\\b/c/d.py')[0], '__pycache__', 'd.cpython-32.pyc')
  'a\\b/c\\__pycache__\\d.cpython-32.pyc

Basically imp.cache_from_source() goes to a bunch of effort to reuse the farthest right separator when there is an alternative separator *before* and path splitting is done. But if you look at ntpath.join(), it doesn't even attempt that much effort.

Now that we can reuse os.path.join() (directly for source_from_cache(), indirectly through easy algorithmic copying in cache_from_source()) do we want to keep the "special" semantics, or can I change it to match what ntpath would do when there can be more than one path separator on an OS (i.e. not do anything special)?

Is there an issue here with importing from zip files, which use / separator, versus importing from the file system, which on Windows can use either / or \ ?  I don't know if imp.cache_from_source cares or is aware, but it is the only thing I can think of that might have an impact on such semantics.  (Well, the other is command line usage, but I don't think you are dealing with command lines at that point.)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: path joining on Windows and imp.cache_from_source()

"Martin v. Löwis"
In reply to this post by Brett Cannon-2
> Now that we can reuse os.path.join() (directly for source_from_cache(),
> indirectly through easy algorithmic copying in cache_from_source()) do
> we want to keep the "special" semantics, or can I change it to match
> what ntpath would do when there can be more than one path separator on
> an OS (i.e. not do anything special)?

This goes back to

http://codereview.appspot.com/842043/diff/1/3#newcode787

where Antoine points out that the code needs to look for altsep.

He then suggests "keep the right-most of both". I don't think he
literally meant that the right-most separator should then also be
used to separate __pycache__, but only that the right-most of
either SEP or ALTSEP is what separates the module name.

In any case, Barry apparently took this comment to mean that the
rightmost separator should be preserved.

So I don't think this is an important feature.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: path joining on Windows and imp.cache_from_source()

Brett Cannon-2
In reply to this post by Glenn Linderman-3


On Sun, Apr 22, 2012 at 01:44, Glenn Linderman <[hidden email]> wrote:
On 4/21/2012 8:53 PM, Brett Cannon wrote:
imp.cache_from_source() (and thus also imp.source_from_cache()) has special semantics compared to how os.path.join() works. For instance, if you look at test_imp you will notice it tries to use the same path separator as is the farthest right in the path it is given::

  self.assertEqual(imp.cache_from_source('\\foo\\bar/baz/qux.py', True), '\\foo\\bar\\baz/__pycache__/qux.{}.pyc'.format(self.tag))

But if you do the same basic operation using ntpath, you will notice it simply doesn't care::

  >>> ntpath.join(ntpath.split('a\\b/c/d.py')[0], '__pycache__', 'd.cpython-32.pyc')
  'a\\b/c\\__pycache__\\d.cpython-32.pyc

Basically imp.cache_from_source() goes to a bunch of effort to reuse the farthest right separator when there is an alternative separator *before* and path splitting is done. But if you look at ntpath.join(), it doesn't even attempt that much effort.

Now that we can reuse os.path.join() (directly for source_from_cache(), indirectly through easy algorithmic copying in cache_from_source()) do we want to keep the "special" semantics, or can I change it to match what ntpath would do when there can be more than one path separator on an OS (i.e. not do anything special)?

Is there an issue here with importing from zip files, which use / separator, versus importing from the file system, which on Windows can use either / or \ ?  I don't know if imp.cache_from_source cares or is aware, but it is the only thing I can think of that might have an impact on such semantics.  (Well, the other is command line usage, but I don't think you are dealing with command lines at that point.)

Right now zipimport doesn't even support __pycache__ (I think). Besides, zipimport already does a string substitution of os.altsep with os.sep (see Modules/zipimport.c:90 amongst other places) so it also doesn't care in the end.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: path joining on Windows and imp.cache_from_source()

Brett Cannon-2
In reply to this post by "Martin v. Löwis"


On Sun, Apr 22, 2012 at 01:45, "Martin v. Löwis" <[hidden email]> wrote:
> Now that we can reuse os.path.join() (directly for source_from_cache(),
> indirectly through easy algorithmic copying in cache_from_source()) do
> we want to keep the "special" semantics, or can I change it to match
> what ntpath would do when there can be more than one path separator on
> an OS (i.e. not do anything special)?

This goes back to

http://codereview.appspot.com/842043/diff/1/3#newcode787

where Antoine points out that the code needs to look for altsep.

He then suggests "keep the right-most of both". I don't think he
literally meant that the right-most separator should then also be
used to separate __pycache__, but only that the right-most of
either SEP or ALTSEP is what separates the module name.

In any case, Barry apparently took this comment to mean that the
rightmost separator should be preserved.

So I don't think this is an important feature.

OK, then I'll go back to ntpath.join()/split() semantics of caring on split about altsep but not on join to keep it consistent w/ os.path and what people are used to.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: path joining on Windows and imp.cache_from_source()

Antoine Pitrou
In reply to this post by "Martin v. Löwis"
On Sun, 22 Apr 2012 07:45:27 +0200
"Martin v. Löwis" <[hidden email]> wrote:

>
> This goes back to
>
> http://codereview.appspot.com/842043/diff/1/3#newcode787
>
> where Antoine points out that the code needs to look for altsep.
>
> He then suggests "keep the right-most of both". I don't think he
> literally meant that the right-most separator should then also be
> used to separate __pycache__, but only that the right-most of
> either SEP or ALTSEP is what separates the module name.

Indeed :-)

Thanks

Antoine.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com