QThread not forcibly terminating as expected

[Brian Knudson]

Hello all,

  Apologies for the lengthy mail.  The last paragraph is the important one, but everything else helps explain.

  I'm writing a PyQt interface for a networking system (3rd party application) via its API.  The 3rd party API calls don't have a configurable timeout.  So that my interface doesn't block, I've made those API calls run in a QThread.  I've attached a timer to the thread & after a certain period of time (10 seconds in this case), I make the thread terminate itself using the very forceful terminate() function (I would happily use another if it would accomplish my goal of stopping execution NOW).  I expect this would kill the API call, but it doesn't seem to.  This happens when the API thinks a host is up, when, in fact, it is not, so it's locked waiting for socket connection/communication, but gets none - until it eventually times itself out some 60 seconds later.

  In my thread class, I log when the run() function starts and when it stops.  I also log when a timeout is reached & the terminate function is called.  What I'm seeing is the run start, the timeout happen (which calls self.terminate()), and a good while later, the end of the run function is logged - which I wouldn't expect.  I would expect the run function to cease doing anything when terminate() is called.  
  I have the thread set to update the UI upon completion.  When my app gets into this state, the UI widget this thread is trying to update is no longer updatable.  The app is still responsive & other widgets will happily update themselves, but this one is locked until the thread finally ends.. at which point it can be updated again, so long as the API returns valid data.

  I have set setTerminationEnabled to True (if that matters).  I'm happy to post the code to my very generic worker thread class if needed

  I wonder if the problem is that the thread is blocked on a function call (the API call that uses sockets)...  Does the API call need to unwind before the thread can terminate?  If that's the case, any suggestions for how to get around this?  

I wouldn't much mind the lack of expected termination if it didn't lock up the widget, but it seems to (which is also surprising).  At the end of the day, that's the important bit.  Terminating the thread is only a way to get the widget to be responsive again sooner.

Thanks,
-Brian
_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt

[Andreas Pakulat]

On 02.03.12 19:59:50, Brian Knudson wrote:
> Hello all,
>
>   Apologies for the lengthy mail.  The last paragraph is the important one, but everything else helps explain.
>
>   I'm writing a PyQt interface for a networking system (3rd party application) via its API.  The 3rd party API calls don't have a configurable timeout.  So that my interface doesn't block, I've made those API calls run in a QThread.  I've attached a timer to the thread & after a certain period of time (10 seconds in this case), I make the thread terminate itself using the very forceful terminate() function (I would happily use another if it would accomplish my goal of stopping execution NOW).  I expect this would kill the API call, but it doesn't seem to.  This happens when the API thinks a host is up, when, in fact, it is not, so it's locked waiting for socket connection/communication, but gets none - until it eventually times itself out some 60 seconds later.

Look at the QThread documentation for terminate(). One almost never
wants to call that function. But it clearly documents that the thread
might or might not terminate immediately.
http://qt-project.org/doc/qt-4.8/qthread.html#terminate

>   In my thread class, I log when the run() function starts and when it stops.  I also log when a timeout is reached & the terminate function is called.  What I'm seeing is the run start, the timeout happen (which calls self.terminate()), and a good while later, the end of the run function is logged - which I wouldn't expect.  I would expect the run function to cease doing anything when terminate() is called.  

See above, you haven't read the API docs ;)

>   I have the thread set to update the UI upon completion.  When my app gets into this state, the UI widget this thread is trying to update is no longer updatable.  The app is still responsive & other widgets will happily update themselves, but this one is locked until the thread finally ends.. at which point it can be updated again, so long as the API returns valid data.

Can you show some sample code for this, I don't quite understand what
you mean with "its not updatable" or "its locked"?

>   I wonder if the problem is that the thread is blocked on a function call (the API call that uses sockets)...  Does the API call need to unwind before the thread can terminate?  If that's the case, any suggestions for how to get around this?  

That would be a question for someone familiar with low-level IO like
sockets.

> I wouldn't much mind the lack of expected termination if it didn't lock up the widget, but it seems to (which is also surprising).  At the end of the day, that's the important bit.  Terminating the thread is only a way to get the widget to be responsive again sooner.

Can you leave out the network-system from your app, replace it with some
kind of busy-loop, i.e.

while(True):
  time.sleep(.5)

Does that still lock up the widget until you terminate the thread? If so
please post a minimal example.

Andreas

_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt

[Brian Knudson]

On Mar 3, 2012, at 4:00 AM, [hidden email] wrote:

Message: 1
Date: Sat, 3 Mar 2012 10:17:39 +0100
From: Andreas Pakulat <[hidden email]>
To: [hidden email]
Subject: Re: [PyQt] QThread not forcibly terminating as expected
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=us-ascii

On 02.03.12 19:59:50, Brian Knudson wrote:

Hello all,

 Apologies for the lengthy mail.  The last paragraph is the important one, but everything else helps explain.

 I'm writing a PyQt interface for a networking system (3rd party application) via its API.  The 3rd party API calls don't have a configurable timeout.  So that my interface doesn't block, I've made those API calls run in a QThread.  I've attached a timer to the thread & after a certain period of time (10 seconds in this case), I make the thread terminate itself using the very forceful terminate() function (I would happily use another if it would accomplish my goal of stopping execution NOW).  I expect this would kill the API call, but it doesn't seem to.  This happens when the API thinks a host is up, when, in fact, it is not, so it's locked waiting for socket connection/communication, but gets none - until it eventually times itself out some 60 seconds later.

Look at the QThread documentation for terminate(). One almost never
wants to call that function. But it clearly documents that the thread
might or might not terminate immediately.
http://qt-project.org/doc/qt-4.8/qthread.html#terminate

I assumed too much, re: "might not terminate immediately".  I took this to mean, "maybe not this instant, but soon," when it appears to mean, "maybe not ever," or better yet, "maybe not as you expect, regardless of timing"

 I have the thread set to update the UI upon completion.  When my app gets into this state, the UI widget this thread is trying to update is no longer updatable.  The app is still responsive & other widgets will happily update themselves, but this one is locked until the thread finally ends.. at which point it can be updated again, so long as the API returns valid data.

Can you show some sample code for this, I don't quite understand what
you mean with "its not updatable" or "its locked"?

As is often the case, while making my example, I found the solution to one of my problems...  I'm using only one thread for this widget, so while it's busy, the widget can't be updated.  Using multiple threads will fix this - to a degree.  Please read on.

I wouldn't much mind the lack of expected termination if it didn't lock up the widget, but it seems to (which is also surprising).  At the end of the day, that's the important bit.  Terminating the thread is only a way to get the widget to be responsive again sooner.

Can you leave out the network-system from your app, replace it with some
kind of busy-loop, i.e.

while(True):
 time.sleep(.5)

Does that still lock up the widget until you terminate the thread? If so
please post a minimal example.

I've made a small example: http://churchofbk.com/misc/qthread_example.py. 

In this example, I've replicated the way I'm doing things in my application so it may not be the ideal way of doing things for this small of an example, but it works.  My question is still related to the fact that threads don't seem to stop executing when I tell them to.

If you run this app, you'll get a dialog with 3 buttons & a text label.  Clicking the buttons will update the text label with some unique string for each button... the string comes from a simulated API call that runs in a thread.  For the outside buttons (button 1 & 3), the text should update instantly, but the center button has a time.sleep(5) before the API return.  All threads have a timeout of 2 seconds.  I expect, then, that button 2 (the center button) will never do anything, as it should timeout before it finishes.  Being that it calls quit (or exit or terminate) before it emits the signal at the end of "run", I expect that the emit will never happen... but it does after the sleep period of 5 seconds.  I expect (maybe incorrectly), that if I click button 1, 2, then 3, quickly in order, I would see the label update for button 1, not for 2, then update for 3 - so at the end, I see button 3's text; but what actually happens is that button 1 updates, button 3 updates, then eventually, button 2 updates - so when everything settles, I'm seeing the result of button 2, even though button 3 was the last button I pressed.

Do I need to keep track of timeout status in the run myself?  In other words, should I be setting some var in __terminator(), then checking the value of that var in run() prior to emitting the signal?

Thanks for taking the time to look at this,

-Brian

_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt

[Andreas Pakulat]

On 05.03.12 13:35:02, Brian Knudson wrote:

> On Mar 3, 2012, at 4:00 AM, [hidden email] wrote:
> >
> > Message: 1
> > Date: Sat, 3 Mar 2012 10:17:39 +0100
> > From: Andreas Pakulat <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [PyQt] QThread not forcibly terminating as expected
> > Message-ID: <[hidden email]>
> > Content-Type: text/plain; charset=us-ascii
> > Can you show some sample code for this, I don't quite understand what
> > you mean with "its not updatable" or "its locked"?
>
> As is often the case, while making my example, I found the solution to one of my problems...  I'm using only one thread for this widget, so while it's busy, the widget can't be updated.  Using multiple threads will fix this - to a degree.  Please read on.

Ah, so with the example you provided there would've been only 1 qthread
object and you re-used it? Thats indeed not a good idea, each thread
should run only once, so you should create a new object each time you
want to run something in a thread.

> > Can you leave out the network-system from your app, replace it with some
> > kind of busy-loop, i.e.
> >
> > while(True):
> >  time.sleep(.5)
> >
> > Does that still lock up the widget until you terminate the thread? If so
> > please post a minimal example.
>
> I've made a small example: http://churchofbk.com/misc/qthread_example.py.
>
> In this example, I've replicated the way I'm doing things in my application so it may not be the ideal way of doing things for this small of an example, but it works.  My question is still related to the fact that threads don't seem to stop executing when I tell them to.

The reason that the thread is not terminating in the code above is that
you're using self.quit(). If you look in the API docs, you'll notice
that quit merely ends the event-loop of the QThread. But in your example
you never start the event-loop in the threads, so there's nothing to
quit. Changing that to use self.terminate() makes the example code work
as expected, the label is not updated for PushButton 2.

If thats not the case for you your Qt and PyQt versions might be
interesting, maybe its a bug in either that got fixed.

Andreas

_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt

[Brian Knudson]

>
> Message: 4
> Date: Tue, 6 Mar 2012 08:54:09 +0100
> From: Andreas Pakulat <[hidden email]>
> To: [hidden email]
> Subject: Re: [PyQt] QThread not forcibly terminating as expected
> Message-ID: <20120306075409.GA30346@barmbek>
> Content-Type: text/plain; charset=us-ascii
>
> On 05.03.12 13:35:02, Brian Knudson wrote:
>> On Mar 3, 2012, at 4:00 AM, [hidden email] wrote:
>>>
>>> Message: 1
>>> Date: Sat, 3 Mar 2012 10:17:39 +0100
>>> From: Andreas Pakulat <[hidden email]>
>>> To: [hidden email]
>>> Subject: Re: [PyQt] QThread not forcibly terminating as expected
>>> Message-ID: <[hidden email]>
>>> Content-Type: text/plain; charset=us-ascii
>>> Can you show some sample code for this, I don't quite understand what
>>> you mean with "its not updatable" or "its locked"?
>>
>> As is often the case, while making my example, I found the solution to one of my problems...  I'm using only one thread for this widget, so while it's busy, the widget can't be updated.  Using multiple threads will fix this - to a degree.  Please read on.
>
> Ah, so with the example you provided there would've been only 1 qthread
> object and you re-used it? Thats indeed not a good idea, each thread
> should run only once, so you should create a new object each time you
> want to run something in a thread.

Good, common sense advice.  Thanks.  Is there any harm in reusing a list of threads (as I've done in the example)?  Is it bad practice?  (I'm somewhat new to this much threading, if you hadn't noticed)

>>> Can you leave out the network-system from your app, replace it with some
>>> kind of busy-loop, i.e.
>>>
>>> while(True):
>>> time.sleep(.5)
>>>
>>> Does that still lock up the widget until you terminate the thread? If so
>>> please post a minimal example.
>>
>> I've made a small example: http://churchofbk.com/misc/qthread_example.py.
>>
>> In this example, I've replicated the way I'm doing things in my application so it may not be the ideal way of doing things for this small of an example, but it works.  My question is still related to the fact that threads don't seem to stop executing when I tell them to.
>
> The reason that the thread is not terminating in the code above is that
> you're using self.quit(). If you look in the API docs, you'll notice
> that quit merely ends the event-loop of the QThread. But in your example
> you never start the event-loop in the threads, so there's nothing to
> quit. Changing that to use self.terminate() makes the example code work
> as expected, the label is not updated for PushButton 2.
>
> If thats not the case for you your Qt and PyQt versions might be
> interesting, maybe its a bug in either that got fixed.

I'd tried that, (as you can see by the fact that it was commented out in the code), but was getting very unexpected results in an earlier incarnation of the example - the application was crashing... not just throwing-an-exception crashing, but an OS level crash.  However, trying it again, as it sits now, seems to work just fine.  Hmm.  I'll go with terminate() & keep an eye on it.

Thanks for all your help.  This has been a good exercise.

-Brian
_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt

[Andreas Pakulat]

On 06.03.12 11:10:19, Brian Knudson wrote:

> >
> > Message: 4
> > Date: Tue, 6 Mar 2012 08:54:09 +0100
> > From: Andreas Pakulat <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [PyQt] QThread not forcibly terminating as expected
> > Message-ID: <20120306075409.GA30346@barmbek>
> > Content-Type: text/plain; charset=us-ascii
> >
> > On 05.03.12 13:35:02, Brian Knudson wrote:
> >> On Mar 3, 2012, at 4:00 AM, [hidden email] wrote:
> >>>
> >>> Message: 1
> >>> Date: Sat, 3 Mar 2012 10:17:39 +0100
> >>> From: Andreas Pakulat <[hidden email]>
> >>> To: [hidden email]
> >>> Subject: Re: [PyQt] QThread not forcibly terminating as expected
> >>> Message-ID: <[hidden email]>
> >>> Content-Type: text/plain; charset=us-ascii
> >>> Can you show some sample code for this, I don't quite understand what
> >>> you mean with "its not updatable" or "its locked"?
> >>
> >> As is often the case, while making my example, I found the solution to one of my problems...  I'm using only one thread for this widget, so while it's busy, the widget can't be updated.  Using multiple threads will fix this - to a degree.  Please read on.
> >
> > Ah, so with the example you provided there would've been only 1 qthread
> > object and you re-used it? Thats indeed not a good idea, each thread
> > should run only once, so you should create a new object each time you
> > want to run something in a thread.
>
> Good, common sense advice.  Thanks.  Is there any harm in reusing a list of threads (as I've done in the example)?  Is it bad practice?  (I'm somewhat new to this much threading, if you hadn't noticed)

I never tried re-starting a QThread, but I believe Qt does not really
support that. If you want to start a thread, create a new object and
start it. Once its terminated or finished, throw it away.

Usually its better to use QThreadPool or the Qt Concurrent framework
(not sure if thats wrapped in PyQt).

> >> I've made a small example: http://churchofbk.com/misc/qthread_example.py.
> >>
> >> In this example, I've replicated the way I'm doing things in my application so it may not be the ideal way of doing things for this small of an example, but it works.  My question is still related to the fact that threads don't seem to stop executing when I tell them to.
> >
> > The reason that the thread is not terminating in the code above is that
> > you're using self.quit(). If you look in the API docs, you'll notice
> > that quit merely ends the event-loop of the QThread. But in your example
> > you never start the event-loop in the threads, so there's nothing to
> > quit. Changing that to use self.terminate() makes the example code work
> > as expected, the label is not updated for PushButton 2.
> >
> > If thats not the case for you your Qt and PyQt versions might be
> > interesting, maybe its a bug in either that got fixed.
>
> I'd tried that, (as you can see by the fact that it was commented out in the code), but was getting very unexpected results in an earlier incarnation of the example - the application was crashing... not just throwing-an-exception crashing, but an OS level crash.  However, trying it again, as it sits now, seems to work just fine.  Hmm.  I'll go with terminate() & keep an eye on it.

Crashes in PyQt are most often either accessing a QObject thats been
deleted from the C++ side already, not keeping a reference to an
Python-created object after handing it to Qt api (which doesn't keep a
reference to the Python-parts and hence the underlying C++ object gets
deleted behind the scenes) or bugs in Qt.

Andreas

_______________________________________________
PyQt mailing list    [hidden email]
http://www.riverbankcomputing.com/mailman/listinfo/pyqt