clean up resources after interrupted page load

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

clean up resources after interrupted page load

Anatoliy Kushner

With sample code below if subscriber interrupt page load somewhere between client.connect and client.close() then cherrypy will hang and will not clean up used resources like: 1) open tcp socket (from netstat -anpt on this server i see tcp connections to server that is established for some days, while time to executing any commands should finish in 10 sec) 2) from ps -p 'cherrypy pid' H , i see , that number of threads increased, and number of threads equals number of unclosed tcp connections

while debugging this i found that thread stop at "yield bytes(a, 'utf-8')" in while loop. So , is there any way to say cherrypy to kill threads after some timeout ?

P.S. response.timeout = 30 already set in config and this does not help



@cherrypy.expose
def test(self, *args, **kwargs):
 
...
 
...
 
def getData(host, cmd):
   html
= open("menuRp1.html").read()
   
yield bytes(html, 'utf-8')
   client
= paramiko.SSHClient()
   client
.connect(hostname=host, username=user, password=pwd, port=22)
   stdin
, stdout, stderr = client.exec_command(cmd)
   a
=stdout.readline()
   
while a:
     
yield bytes(a, 'utf-8')
     a
=stdout.readline()
   client
.close()
   html
= open("menuRp2.html").read()
   
yield bytes(html, 'utf-8')
 
return getData(h, c)
test
._cp_config = {'response.stream': True}


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/92d0f922-5e40-4a5c-b3d6-a99d10ff0830%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clean up resources after interrupted page load

Sviatoslav Sydorenko-2
Timeout feature has been deprecated because it's not really possible to cancel threads. The solution would be to rewrite the engine to use async paradigm.

Try submitting an issue against Cheroot with a minimal reproducer. There's some WIP there which aims to improve the connection management.


--Sviatoslav.

Sent from my phone, please pardon any typos.

сб, 29 черв. 2019, 10:05 користувач Anatoliy Kushner <[hidden email]> пише:

With sample code below if subscriber interrupt page load somewhere between client.connect and client.close() then cherrypy will hang and will not clean up used resources like: 1) open tcp socket (from netstat -anpt on this server i see tcp connections to server that is established for some days, while time to executing any commands should finish in 10 sec) 2) from ps -p 'cherrypy pid' H , i see , that number of threads increased, and number of threads equals number of unclosed tcp connections

while debugging this i found that thread stop at "yield bytes(a, 'utf-8')" in while loop. So , is there any way to say cherrypy to kill threads after some timeout ?

P.S. response.timeout = 30 already set in config and this does not help



@cherrypy.expose
def test(self, *args, **kwargs):
 
...
 
...
 
def getData(host, cmd):
   html
= open("menuRp1.html").read()
   
yield bytes(html, 'utf-8')
   client
= paramiko.SSHClient()
   client
.connect(hostname=host, username=user, password=pwd, port=22)
   stdin
, stdout, stderr = client.exec_command(cmd)
   a
=stdout.readline()
   
while a:
     
yield bytes(a, 'utf-8')
     a
=stdout.readline()
   client
.close()
   html
= open("menuRp2.html").read()
   
yield bytes(html, 'utf-8')
 
return getData(h, c)
test
._cp_config = {'response.stream': True}


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/92d0f922-5e40-4a5c-b3d6-a99d10ff0830%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAFYONRBoAZFtBsPj1xs3GzLDac5iCvww5zd_wVbEPs4gVs_pGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clean up resources after interrupted page load

Robert Brewer-2
In Python, you have to explicitly cancel threads inside the thread. So one thing you _can_ do is add a line inside your while loop: "if response.timed_out: raise cherrypy.TimeoutError()". It won't do anything if the thread blocks forever on readline(), but it will allow the thread to exit as long as readline is yielding.

On Sat, Jun 29, 2019, 11:25 Sviatoslav Sydorenko <[hidden email]> wrote:
Timeout feature has been deprecated because it's not really possible to cancel threads. The solution would be to rewrite the engine to use async paradigm.

Try submitting an issue against Cheroot with a minimal reproducer. There's some WIP there which aims to improve the connection management.


--Sviatoslav.

Sent from my phone, please pardon any typos.

сб, 29 черв. 2019, 10:05 користувач Anatoliy Kushner <[hidden email]> пише:

With sample code below if subscriber interrupt page load somewhere between client.connect and client.close() then cherrypy will hang and will not clean up used resources like: 1) open tcp socket (from netstat -anpt on this server i see tcp connections to server that is established for some days, while time to executing any commands should finish in 10 sec) 2) from ps -p 'cherrypy pid' H , i see , that number of threads increased, and number of threads equals number of unclosed tcp connections

while debugging this i found that thread stop at "yield bytes(a, 'utf-8')" in while loop. So , is there any way to say cherrypy to kill threads after some timeout ?

P.S. response.timeout = 30 already set in config and this does not help



@cherrypy.expose
def test(self, *args, **kwargs):
 
...
 
...
 
def getData(host, cmd):
   html
= open("menuRp1.html").read()
   
yield bytes(html, 'utf-8')
   client
= paramiko.SSHClient()
   client
.connect(hostname=host, username=user, password=pwd, port=22)
   stdin
, stdout, stderr = client.exec_command(cmd)
   a
=stdout.readline()
   
while a:
     
yield bytes(a, 'utf-8')
     a
=stdout.readline()
   client
.close()
   html
= open("menuRp2.html").read()
   
yield bytes(html, 'utf-8')
 
return getData(h, c)
test
._cp_config = {'response.stream': True}


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/92d0f922-5e40-4a5c-b3d6-a99d10ff0830%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAFYONRBoAZFtBsPj1xs3GzLDac5iCvww5zd_wVbEPs4gVs_pGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAJ4gti-fbZrzM-mfy7_OKDuTqBVLCJWWZ11LKOr9eB6d3tomDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: clean up resources after interrupted page load

Sviatoslav Sydorenko-2
It seems to be the case that it blocks on socket interactions...

This will supposedly improve the situation https://github.com/cherrypy/cheroot/pull/176


--Sviatoslav.

Sent from my phone, please pardon any typos.

сб, 29 черв. 2019, 20:34 користувач Robert Brewer <[hidden email]> пише:
In Python, you have to explicitly cancel threads inside the thread. So one thing you _can_ do is add a line inside your while loop: "if response.timed_out: raise cherrypy.TimeoutError()". It won't do anything if the thread blocks forever on readline(), but it will allow the thread to exit as long as readline is yielding.

On Sat, Jun 29, 2019, 11:25 Sviatoslav Sydorenko <[hidden email]> wrote:
Timeout feature has been deprecated because it's not really possible to cancel threads. The solution would be to rewrite the engine to use async paradigm.

Try submitting an issue against Cheroot with a minimal reproducer. There's some WIP there which aims to improve the connection management.


--Sviatoslav.

Sent from my phone, please pardon any typos.

сб, 29 черв. 2019, 10:05 користувач Anatoliy Kushner <[hidden email]> пише:

With sample code below if subscriber interrupt page load somewhere between client.connect and client.close() then cherrypy will hang and will not clean up used resources like: 1) open tcp socket (from netstat -anpt on this server i see tcp connections to server that is established for some days, while time to executing any commands should finish in 10 sec) 2) from ps -p 'cherrypy pid' H , i see , that number of threads increased, and number of threads equals number of unclosed tcp connections

while debugging this i found that thread stop at "yield bytes(a, 'utf-8')" in while loop. So , is there any way to say cherrypy to kill threads after some timeout ?

P.S. response.timeout = 30 already set in config and this does not help



@cherrypy.expose
def test(self, *args, **kwargs):
 
...
 
...
 
def getData(host, cmd):
   html
= open("menuRp1.html").read()
   
yield bytes(html, 'utf-8')
   client
= paramiko.SSHClient()
   client
.connect(hostname=host, username=user, password=pwd, port=22)
   stdin
, stdout, stderr = client.exec_command(cmd)
   a
=stdout.readline()
   
while a:
     
yield bytes(a, 'utf-8')
     a
=stdout.readline()
   client
.close()
   html
= open("menuRp2.html").read()
   
yield bytes(html, 'utf-8')
 
return getData(h, c)
test
._cp_config = {'response.stream': True}


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/92d0f922-5e40-4a5c-b3d6-a99d10ff0830%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAFYONRBoAZFtBsPj1xs3GzLDac5iCvww5zd_wVbEPs4gVs_pGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAJ4gti-fbZrzM-mfy7_OKDuTqBVLCJWWZ11LKOr9eB6d3tomDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/CAFYONRA%3Dv_Nib2%3DHcm00N2pWoo0%2BAqLF5TZDtMpzHu9Q-ThuOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.