Process Data during server initialization

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Process Data during server initialization

Lois Greene-Hernandez
Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Process Data during server initialization

Scott Sawyer
Could you run a cron job on the system to analyze the data periodically?

Sent from my iPhone

On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez <[hidden email]> wrote:

Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Process Data during server initialization

Lois Greene-Hernandez
Well it's data that I need to populate the pages.  It's very processor intensive and I'll need to pass it back to the views once it's processed.  Can I pass data back to the views once I've run a task?  I'm looking into caching alternatives.  

Thanks
Lois

On Thu, Oct 22, 2020 at 4:55 PM Scott Sawyer <[hidden email]> wrote:
Could you run a cron job on the system to analyze the data periodically?

Sent from my iPhone

On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez <[hidden email]> wrote:

Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/X1joo7QEbnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAAGmcw0tk9fd%3DhjvpdZTpqqf9-gDDKgtW7YYfvJxVcCo1DvD4g%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Process Data during server initialization

Okware Aldo
Hi Lois, 

To expand on what Scott is saying. 
- model DB tables to store the data you need to render the graphs
- build a service to preprocess the data - this service can run periodically depending on how up to date you need the graphs to be (a microservice)
- build a single endpoint to simple fetch the data the graphs need (Django) if this is a monolith the normal Django views should do the trick






On Fri, Oct 23, 2020 at 12:01 AM Lois Greene-Hernandez <[hidden email]> wrote:
Well it's data that I need to populate the pages.  It's very processor intensive and I'll need to pass it back to the views once it's processed.  Can I pass data back to the views once I've run a task?  I'm looking into caching alternatives.  

Thanks
Lois

On Thu, Oct 22, 2020 at 4:55 PM Scott Sawyer <[hidden email]> wrote:
Could you run a cron job on the system to analyze the data periodically?

Sent from my iPhone

On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez <[hidden email]> wrote:

Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/X1joo7QEbnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAAGmcw0tk9fd%3DhjvpdZTpqqf9-gDDKgtW7YYfvJxVcCo1DvD4g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMEZma-P3Z6yoAJJGCWbDpd8YEWkc_kV8uS4FKDNvsR-LF-4WA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Process Data during server initialization

Shaheed Haque-2
Well, it's a bit of a blunt weapon, but Python is going to run any code it finds at module scope as the process is initialised. So you could cause your heavy stuff to be run there? Or at least spawned from there? 

On Thu, 22 Oct 2020, 22:59 Okware Aldo, <[hidden email]> wrote:
Hi Lois, 

To expand on what Scott is saying. 
- model DB tables to store the data you need to render the graphs
- build a service to preprocess the data - this service can run periodically depending on how up to date you need the graphs to be (a microservice)
- build a single endpoint to simple fetch the data the graphs need (Django) if this is a monolith the normal Django views should do the trick






On Fri, Oct 23, 2020 at 12:01 AM Lois Greene-Hernandez <[hidden email]> wrote:
Well it's data that I need to populate the pages.  It's very processor intensive and I'll need to pass it back to the views once it's processed.  Can I pass data back to the views once I've run a task?  I'm looking into caching alternatives.  

Thanks
Lois

On Thu, Oct 22, 2020 at 4:55 PM Scott Sawyer <[hidden email]> wrote:
Could you run a cron job on the system to analyze the data periodically?

Sent from my iPhone

On Oct 22, 2020, at 1:30 PM, Lois Greene-Hernandez <[hidden email]> wrote:

Hello, 

I'm working on a data and calculation intensive django application.  It uses pandas and plotly.py to create many charts and graphs.  I have some control over the design of the application but not total control.  One decision that I stuck with is that all charts and graphs are preprocessed on the first page that loads.  If it were up to me, I'd process what is needed at the time but its not up to me.  

Since I need to load and process a lot of data prior to page load, I was wondering if it would be possible to do this during server start up.  I'd also need to pass a log of data from whatever startup process I ran to the django views.  

Any information that you could shed on this problem would be greatly appreciated. 

Thanks

Lois J. Greene-Hernandez 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34166f-ade9-4dec-bb9d-0a8d6392fdc8n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/X1joo7QEbnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/F90610CE-721D-4FEB-8992-FAF63A6C6368%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAAGmcw0tk9fd%3DhjvpdZTpqqf9-gDDKgtW7YYfvJxVcCo1DvD4g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAMEZma-P3Z6yoAJJGCWbDpd8YEWkc_kV8uS4FKDNvsR-LF-4WA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHAc2jdi4d_Rz4X4Kue-CbB%2B88-EQyD5dcXqxSysKvBB9hFXFA%40mail.gmail.com.