[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor

New submission from Fabian Groffen <[hidden email]>:

With current working dir an NFS-mounted ZFS share, and /var/tmp (OSX default) HFS+:

% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 24 2012, 19:33:45)
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>>
% ls /var/tmp/testfile ./testfile
./testfile  /var/tmp/testfile

The problem likely is that the flags stored on the HFS+ volume cannot be applied to the NFS-mounted ZFS volume.  This likely also occurs when doing a bit more regular things, like e.g. moving/copying to a mounted USB disk (with FAT32 filesystem).

I believe this is a "regression" introduced by http://bugs.python.org/issue8746.  Python-2.7.2 works fine.

While preserving flags is nice, it is questionable whether failure to do so in this case is worth dying for.  In particular, leaving behind both the original as well as the copy is a bit messy.

----------
components: None
messages: 159178
nosy: grobian
priority: normal
severity: normal
status: open
title: shutil.move broken in 2.7.3 on OSX (chflags fails)
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

I guess a “best effort” approach would be best here.

I presume Python 3.2+ have the same behavior?

----------
nosy: +hynek

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Fabian Groffen <[hidden email]> added the comment:

> I presume Python 3.2+ have the same behavior?

I cannot compile that or get it working normally, so I can't tell for sure.  Judging from the code, I'd say yes.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

Now that’s odd. I just looked into the code at http://hg.python.org/cpython/file/2.7/Lib/shutil.py#l103 and there is a guard against EOPNOTSUPP:

try:
   os.chflags(dst, st.st_flags)
except OSError, why:
   if (not hasattr(errno, 'EOPNOTSUPP') or
   why.errno != errno.EOPNOTSUPP):
      raise

Does your /Library/Gentoo/usr/lib/python2.7/shutil.py look the same? Would you mind adding a `print why.errno` just before the raise?

I have tried move'ing files to NTFS and FAT32 and it works just fine here.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Fabian Groffen <[hidden email]> added the comment:

% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37)
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>>
% vi /Library/Gentoo/usr/lib/python2.7/shutil.py
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37)
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
45
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 300, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 130, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>>
% grep 45 /usr/include/sys/errno.h
#define ENOTSUP         45              /* Operation not supported */

I tried with a FAT16 formatted USB-disk, but there it doesn't fail, so I did some further digging.  MS-DOS FS (under OSX) just seems to support setting flags (I tried with stat.UF_HIDDEN, Finder no longer displays the file).

NFS, however, does NOT support any chflags call.

% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37)
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import errno
>>> print hasattr(errno, 'EOPNOTSUPP')
True
>>> print errno.EOPNOTSUPP
102
>>>

102 obviously != 45

% grep 102 /usr/include/sys/errno.h
#define EOPNOTSUPP      102             /* Operation not supported on socket */

I believe Python got it mixed up here, we're looking for ENOTSUP, but that one doesn't exist, at least not here.

>>> print hasattr(errno, 'ENOTSUP')
False

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Fabian Groffen <[hidden email]> added the comment:

it seems errnomodule.c has no idea of ENOTSUP, and that's not the only missing one.

OSX 10.7:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ENOTSUP
missing: EBADRPC
missing: ERPCMISMATCH
missing: EPROGUNAVAIL
missing: EPROGMISMATCH
missing: EPROCUNAVAIL
missing: EFTYPE
missing: EAUTH
missing: ENEEDAUTH
missing: EPWROFF
missing: EDEVERR
missing: EBADEXEC
missing: EBADARCH
missing: ESHLIBVERS
missing: EBADMACHO
missing: ECANCELED
missing: ENOATTR
missing: ENOPOLICY
missing: ENOTRECOVERABLE
missing: EOWNERDEAD
missing: ELAST

Solaris 10:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ECANCELED
missing: ENOTSUP
missing: EOWNERDEAD
missing: ENOTRECOVERABLE
missing: ELOCKUNMAPPED
missing: ENOTACTIVE

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Ned Deily <[hidden email]> added the comment:

Thanks for the analysis.  Yes, it looks like there's a difference between OS X and current FreeBSDs, for example.  chflags(2) on the latter is documented as returning EOPNOTSUPP and on the former ENOTSUP. shutil should check for both.  A quick search of the source tree did not find any other users of chflags in the standard library. As far as adding other missing errnos, that could be handled as a separate issue as it more of an enhancement.  Anyone interested in contributing a patch for either or both?

https://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man2/chflags.2.html

http://www.freebsd.org/cgi/man.cgi?query=chflags&apropos=0&sektion=2&manpath=FreeBSD+9.0-RELEASE&arch=default&format=html

----------
nosy: +ned.deily
stage:  -> needs patch
versions: +Python 3.2, Python 3.3

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

I had this text ready before ned chimed in, I’ll post it anyway because it was a lot of work ;):

You're right, 2.7’s errnos are incomplete compared to 3.2. Antoine added ENOTSUP in c370866f30a7 and it runs as “Solaris-specific”.

So it’s currently in 3.2 and later. Shouldn’t hurt to back port it?

EOPNOTSUP is obviously wrong in your case and it doesn’t really sound right at all by the description. However, maybe on some other architecture (FreeBSD?) it’s the way to go? The commit (2e0d58adadbe) states it’s because of ZFS on OS X.

As the code is unchanged in 3.2+, this bug also applies to them.


Suggestion:

For 3.2+3.3: I’d extend the catch to also catch ENOTSUP
For 2.7: I’d also backport the err code.

NB I’m fine if Fabian wants to do it himself, it’s his issue.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Fabian Groffen <[hidden email]> added the comment:

I don't want to go through the paperwork nonsense just for a trivial patch, hence I didn't supply one, but instead provided all the information for you guys to make the correct fix.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Éric Araujo <[hidden email]> added the comment:

Trivial patches don’t require paperwork; non-trivial patches require a simple contributor agreement (print, sign, scan, email).  We don’t like that either but it is required.  If you have any suggestion to make the process simpler, please share them on python-dev.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

This is a fix for 2.7.

As Benjamin said in http://bugs.python.org/issue14682#msg159477 it’s okay to back port ENOTSUP, I did it as part of the patch here. I wasn’t sure whether we should document it?

I’m porting the patch to tip right now, reviews/opinions welcome.

----------
keywords: +patch
nosy: +pitrou
Added file: http://bugs.python.org/file25382/expand-chflags-catch-2.7.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

This one is against tip.

----------
Added file: http://bugs.python.org/file25383/expand-chflags-catch-tip.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Hynek Schlawack <[hidden email]> added the comment:

And finally against 3.2

----------
Added file: http://bugs.python.org/file25384/expand-chflags-catch-3.2.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Changes by Hynek Schlawack <[hidden email]>:


----------
components: +Library (Lib) -None
stage: needs patch -> patch review

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move broken in 2.7.3 on OSX (chflags fails)

STINNER Victor
In reply to this post by STINNER Victor

Antoine Pitrou <[hidden email]> added the comment:

> I wasn’t sure whether we should document it?

No, it should remain "hidden".

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14662] shutil.move doesn't handle ENOTSUP raised by chflags on OS X

STINNER Victor
In reply to this post by STINNER Victor

Changes by Hynek Schlawack <[hidden email]>:


----------
assignee:  -> hynek
title: shutil.move broken in 2.7.3 on OSX (chflags fails) -> shutil.move doesn't handle ENOTSUP raised by chflags on OS X

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14662>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Loading...