This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to:
has no problem.
But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug.
I believe I'm still encountering this issue in 2.7:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run
File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted.
There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file. In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__. So I am not sure how much good it would do to fix just one example.
Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception. I am not sure whether you would want to see a backport of this. Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang.