Error in python multiprocessing!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Error in python multiprocessing!

Nederlands mailing list
Hi all,

After a lot of searching on the internet, I am looking for help from you guys.

I am working on a Qt based GUI which takes in a stream of file paths coming from a separate process and reads those data file for post-processing and plotting. Since post-processing and plotting have become a bit involved, there is a request to improve the performance of the GUI. Currently the GUI takes in the input message with the path of the file and gives a SIGNAL to the Worker thread (Qt thread) to process the file. The processing results are saved in the object for reference and display within the GUI. Simple separation of the threads does not help a lot for intensive processing cases. I had to add a sleep of 1 sec to quick fix the GIL of Python and make the GUI responsive. To improve the solution, I thought of using Multiprocessing for the post-processing task. I am trying to add the task to queue and get back the result in a queue. When I implement this solution, I get 'TypeError: cannot serailize '_io.TextIOWrapper' object' at the call of start() of the Process.

What could be the cause of this error? Is there a better way to solve this issue? Any suggestions are welcome.
 
Regards, Akshay Verma

_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error in python multiprocessing!

Martin Fitzpatrick
Hi Akshay

It's a little difficult to understand what you're doing, but as a general rule I would avoid using both multiprocessing and Qt threads (if only because this is going to confuse things). The error you're receiving looks like it is coming from the serialisation (pickle?) of the data in/out of the task. 

For a problem like this I would suggest sticking with Qt threads and implementing something via QRunnable/QThreadpool. I have a brief (draft) writeup of this process on my site at: http://mfitzp.io/application-multithreading-with-pyqt/  ...will be finished up in a few weeks.

The approach is to create a QRunnable class that accepts a function to run and passes notification/results out as signals. To start a job (e.g. process a file) you would pass in your function, the filename and then attach a listener to the 'result' signal. Once the task is run your data will come back out via that. You can for example attach this signal to a slot (python function) to update your GUI.

If the plotting is causing a slow down, you might also want to try PyQtGraph.

Hope that helps

Martin



On 27 August 2016 at 10:16, Akshay Verma via Python-nl <[hidden email]> wrote:
Hi all,

After a lot of searching on the internet, I am looking for help from you guys.

I am working on a Qt based GUI which takes in a stream of file paths coming from a separate process and reads those data file for post-processing and plotting. Since post-processing and plotting have become a bit involved, there is a request to improve the performance of the GUI. Currently the GUI takes in the input message with the path of the file and gives a SIGNAL to the Worker thread (Qt thread) to process the file. The processing results are saved in the object for reference and display within the GUI. Simple separation of the threads does not help a lot for intensive processing cases. I had to add a sleep of 1 sec to quick fix the GIL of Python and make the GUI responsive. To improve the solution, I thought of using Multiprocessing for the post-processing task. I am trying to add the task to queue and get back the result in a queue. When I implement this solution, I get 'TypeError: cannot serailize '_io.TextIOWrapper' object' at the call of start() of the Process.

What could be the cause of this error? Is there a better way to solve this issue? Any suggestions are welcome.
 
Regards, Akshay Verma

_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl



_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error in python multiprocessing!

Guido Wesdorp-3
Hi there,

Seems like Martin is right, the error seems to be that your object isn't serializable, and it indeed seems to be 'pickle' that's used for serialization (though I'm not 100% sure). The _io.TextIOWrapper you're trying to queue is probably a 'foreign' object, implemented in C or C++ or sth, and can therefore not be serialized. A solution would be to store your data in a simpler structure such as a dict or a simple object and add that to the queue instead.

Hope this helps.

Cheers,

Guido

On Sat, Sep 3, 2016 at 9:02 PM, Martin Fitzpatrick <[hidden email]> wrote:
Hi Akshay

It's a little difficult to understand what you're doing, but as a general rule I would avoid using both multiprocessing and Qt threads (if only because this is going to confuse things). The error you're receiving looks like it is coming from the serialisation (pickle?) of the data in/out of the task. 

For a problem like this I would suggest sticking with Qt threads and implementing something via QRunnable/QThreadpool. I have a brief (draft) writeup of this process on my site at: http://mfitzp.io/application-multithreading-with-pyqt/  ...will be finished up in a few weeks.

The approach is to create a QRunnable class that accepts a function to run and passes notification/results out as signals. To start a job (e.g. process a file) you would pass in your function, the filename and then attach a listener to the 'result' signal. Once the task is run your data will come back out via that. You can for example attach this signal to a slot (python function) to update your GUI.

If the plotting is causing a slow down, you might also want to try PyQtGraph.

Hope that helps

Martin



On 27 August 2016 at 10:16, Akshay Verma via Python-nl <[hidden email]> wrote:
Hi all,

After a lot of searching on the internet, I am looking for help from you guys.

I am working on a Qt based GUI which takes in a stream of file paths coming from a separate process and reads those data file for post-processing and plotting. Since post-processing and plotting have become a bit involved, there is a request to improve the performance of the GUI. Currently the GUI takes in the input message with the path of the file and gives a SIGNAL to the Worker thread (Qt thread) to process the file. The processing results are saved in the object for reference and display within the GUI. Simple separation of the threads does not help a lot for intensive processing cases. I had to add a sleep of 1 sec to quick fix the GIL of Python and make the GUI responsive. To improve the solution, I thought of using Multiprocessing for the post-processing task. I am trying to add the task to queue and get back the result in a queue. When I implement this solution, I get 'TypeError: cannot serailize '_io.TextIOWrapper' object' at the call of start() of the Process.

What could be the cause of this error? Is there a better way to solve this issue? Any suggestions are welcome.
 
Regards, Akshay Verma

_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl



_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl



_______________________________________________
Python-nl mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/python-nl
Loading...