FOR LOOP PERFORMANCE

Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

FOR LOOP PERFORMANCE

Abhishek_Gupta
Hello Group members, 

I have been using cherrypy + tubogears framework for around 4 years front ending a product.
Recently, I have been assigned with the task for cloud migration & load testing.

I have noticed that sometimes main URL which returns 200 topmost records from MySQL DB performs inconsistently.
URL performs following steps:
Step 1: DB Query for 200 records. (~0.1 sec)
Step 2: FOR loop over 200 records from STEP 1 & do some changes to the data for presentation.
             Performance of this step is inconsistent. As user load picks up, the performance degrades.

Application stack: Turbogears, cherrypy, SQLAlchemy, MySQL


Please refer to timings which were captured in application logs.

2018-09-26 09:57:08,657-[CRITICAL] <<071b00a89eca7e090ddf15081c14ead17b851d9110.199.123.53>> FOR LOOP TOOK 13.0 sec.
2018-09-26 09:57:08,660-[CRITICAL] <<071b00a89eca7e090ddf15081c14ead17b851d9110.199.123.53>> DISPLAYING 62 RESULTS TOOK 13.1690001488 sec.

2018-09-26 10:01:52,174-[CRITICAL]  <<76ef224e3c054020a1fbaa258f2dec20c4be3ae610.199.120.100>> FOR LOOP TOOK 17.4279999733 sec.
2018-09-26 10:01:52,176-[CRITICAL]  <<76ef224e3c054020a1fbaa258f2dec20c4be3ae610.199.120.100>> DISPLAYING 200 RESULTS TOOK 17.628000021 sec.

2018-09-26 10:06:15,165-[CRITICAL] <<e3251a737eaf0128e6cb0321a3f51a91038bdfaf10.199.140.69>> FOR LOOP TOOK 5.58699989319 sec
2018-09-26 10:06:15,167-[CRITICAL] <<e3251a737eaf0128e6cb0321a3f51a91038bdfaf10.199.140.69>> DISPLAYING 57 RESULTS TOOK 5.65600013733 sec.

2018-09-26 10:26:31,381-[CRITICAL] <<e3251a737eaf0128e6cb0321a3f51a91038bdfaf10.199.140.69>> FOR LOOP TOOK 5.21700000763 sec.
2018-09-26 10:26:31,381-[CRITICAL]  <<e3251a737eaf0128e6cb0321a3f51a91038bdfaf10.199.140.69>> DISPLAYING 38 RESULTS TOOK 5.29600000381 sec.

2018-09-26 10:01:09,368-[CRITICAL] <<169ef276f8f79c3728f90b8439ed1007f13337d5192.168.47.69>> FOR LOOP TOOK 6.16900014877 sec
2018-09-26 10:01:09,368-[CRITICAL] <<169ef276f8f79c3728f90b8439ed1007f13337d5192.168.47.69>> DISPLAYING 20 RESULTS TOOK 6.39199995995 sec.





    

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: FOR LOOP PERFORMANCE

Tim Roberts
abhishek wrote:

>
> I have been using cherrypy + tubogears framework for around 4 years
> front ending a product.
> Recently, I have been assigned with the task for cloud migration &
> load testing.
>
> I have noticed that sometimes main URL which returns 200 topmost
> records from MySQL DB performs inconsistently.
> URL performs following steps:
> Step 1: DB Query for 200 records. (~0.1 sec)
> Step 2: FOR loop over 200 records from STEP 1 & do some changes to the
> data for presentation.
>              Performance of this step is inconsistent. As user load
> picks up, the performance degrades.
Your statistics don't show varying times for 200 records; each of these
queries is for a different size.  The time for 200 results is roughly 4
times as long as the time for 57 results, so that doesn't seem
particularly outrageous.  However, it's hard for me to imagine what
would take 17 seconds to display 200 results.  Are you doing more
database queries for each row?

--
Tim Roberts, [hidden email]
Providenza & Boekelheide, Inc.


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FOR LOOP PERFORMANCE

Michiel Overtoom
In reply to this post by Abhishek_Gupta
Hi,

> On 2018-09-28, at 18:43, abhishek <[hidden email]> wrote:
>
> ...do some changes to the data for presentation...

It sounds like this part is horribly inefficient. Can you show the code which does these 'changes'? Are you using membership tests in lists instead of sets/dict, perhaps? Are you creating very heavy objects? Deeply nested loops?

Greetings,

--
"You can't actually make computers run faster, you can only make them do less." - RiderOfGiraffes

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.