Quantcast

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor

New submission from Nikolaus Rath <[hidden email]>:

The attached test program calls apply_async with a function that will raise CalledProcessError. However, when result.get() is called, it raises a TypeError and the program hangs:

$ ./bug.py
ERROR:root:ops
Traceback (most recent call last):
  File "./bug.py", line 19, in run_with
    dW1 = run_dcon()
  File "./bug.py", line 26, in run_dcon
    subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
  File "/usr/lib/python2.6/subprocess.py", line 498, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['dcon']' returned non-zero exit status 127
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())

----------
components: Library (Lib)
files: bug.py
messages: 111827
nosy: Nikratio
priority: normal
severity: normal
status: open
title: multiprocessing.pool.AsyncResult.get() messes up exceptions
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file18233/bug.py

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor

Changes by R. David Murray <[hidden email]>:


----------
nosy: +jnoller
stage:  -> unit test needed

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Ray.Allen <[hidden email]> added the comment:

I got "OSError: [Errno 2] No such file or directory" instead of TypeError when running the "bug.py", did I miss something?

----------
nosy: +ysj.ray

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Jesse Noller <[hidden email]> added the comment:

@ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Nikolaus Rath <[hidden email]> added the comment:

@ray: Try it with the following dummy dcon program:

$ cat dcon
#!/bin/sh

exit 127

(and change the path to dcon in bug.py accordingly).

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Ray.Allen <[hidden email]> added the comment:

This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to:
          self.__init__(*self.args)
has no problem.

But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug.

----------
keywords: +patch
Added file: http://bugs.python.org/file18309/issue9400.diff

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Daniel Blanchard <[hidden email]> added the comment:

I believe I'm still encountering this issue in 2.7:


Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())

----------
nosy: +Daniel.Blanchard
versions: +Python 2.7 -Python 2.6

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Amaury Forgeot d'Arc <[hidden email]> added the comment:

Here is another patch, with test.

----------
nosy: +amaury.forgeotdarc
stage: test needed -> patch review
Added file: http://bugs.python.org/file25111/picklable_process_exception.patch

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Daniel Blanchard <[hidden email]> added the comment:

The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Changes by Antoine Pitrou <[hidden email]>:


----------
nosy: +sbt

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

STINNER Victor
In reply to this post by STINNER Victor

Richard Oudkerk <[hidden email]> added the comment:

There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file.  In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__.  So I am not sure how much good it would do to fix just one example.

Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception.  I am not sure whether you would want to see a backport of this.  Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue9400>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Loading...