Asynchronous web tasks

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Asynchronous web tasks

Meenal Pant-2
I use Django in production and now have a need to perform asynchronous backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?

I am looking for some opinions /recommendations to use one of  these two technologies or something else altogether ?
Thanks
Meenal

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Rami Chowdhury
Hi Meenal,

On Feb 2, 2012, at 4:28 PM, Meenal Pant wrote:

I use Django in production and now have a need to perform asynchronous backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

What sort of response is needed to the user -- is this just something the user triggers and then forgets about, or do you want to show the user some kind of "loading" view and then respond to them when the job is done? 


I have been thinking of using twisted or message queues (RabbitMQ and celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?

I am looking for some opinions /recommendations to use one of  these two technologies or something else altogether ?
Thanks
Meenal
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies

----
Rami Chowdhury
"A mind all logic is like a knife all blade - it makes the hand bleed that uses it." -- Rabindranath Tagore
+44-7581-430-517


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Dirk Bergstrom-4
In reply to this post by Meenal Pant-2
On 02/02/2012 08:28 AM, Meenal Pant wrote:
> I use Django in production and now have a need to perform asynchronous
> backend tasks that
> a. Are some scheduled jobs
> b. Jobs triggered via a user input via Django frontend.
> I have been thinking of using twisted or message queues (RabbitMQ and
> celery) to achieve this.  From my research so far:
> I am looking for some opinions /recommendations to use one of  these two
> technologies or something else altogether ?

How's about cron for a and subprocess.Popen for b?

Start with the simplest implementation that will work, and only move to
something complicated if you desperately need it.  Adding another
heavyweight set of libraries to a Django app is not going to help you
get more sleep at night.

--
Dirk Bergstrom
[hidden email]
http://otisbean.com/
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Tung Wai Yip
In reply to this post by Meenal Pant-2
I'm using RabbitMQ and Celery. It certainly can be triggered by both  
cases. In addition, you can deploy it on different servers if necessary.

Wai Yip

> I use Django in production and now have a need to perform asynchronous
> backend tasks that
> a. Are some scheduled jobs
> b. Jobs triggered via a user input via Django frontend.
>
> I have been thinking of using twisted or message queues (RabbitMQ and
> celery) to achieve this.  From my research so far:
>
> 1. Twisted allows both a and b though the learning curve with twisted is
> steeper
> 2. RabbitMQ and celery is easier to use, allows a but does it also allow  
> b?
>
> I am looking for some opinions /recommendations to use one of  these two
> technologies or something else altogether ?
> Thanks
> Meenal
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Shannon -jj Behrens
Celery is very popular for this sort of stuff.  There was a talk on it at the last PyCon.

-jj

On Thu, Feb 2, 2012 at 9:36 AM, Tung Wai Yip <[hidden email]> wrote:
I'm using RabbitMQ and Celery. It certainly can be triggered by both cases. In addition, you can deploy it on different servers if necessary.

Wai Yip


I use Django in production and now have a need to perform asynchronous
backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and
celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is
steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?

I am looking for some opinions /recommendations to use one of  these two
technologies or something else altogether ?
Thanks
Meenal
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies



--
In this life we cannot do great things. We can only do small things with great love. -- Mother Teresa

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

joshua kaderlan-2
In reply to this post by Dirk Bergstrom-4
Yeah, this is the standard advice I've seen for solving this problem: cron for scheduled jobs and RabbitMQ/celery or 0MQ for other tasks.



From: Dirk Bergstrom <[hidden email]>
To: [hidden email]
Sent: Thursday, February 2, 2012 9:19 AM
Subject: Re: [Baypiggies] Asynchronous web tasks

On 02/02/2012 08:28 AM, Meenal Pant wrote:
> I use Django in production and now have a need to perform asynchronous
> backend tasks that
> a. Are some scheduled jobs
> b. Jobs triggered via a user input via Django frontend.
> I have been thinking of using twisted or message queues (RabbitMQ and
> celery) to achieve this.  From my research so far:
> I am looking for some opinions /recommendations to use one of  these two
> technologies or something else altogether ?

How's about cron for a and subprocess.Popen for b?

Start with the simplest implementation that will work, and only move to something complicated if you desperately need it.  Adding another heavyweight set of libraries to a Django app is not going to help you get more sleep at night.

-- Dirk Bergstrom
[hidden email]
http://otisbean.com/
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies



_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Ryan Matthew Balfanz
In reply to this post by Tung Wai Yip
I meant to reply sooner, and a few have already suggested celery, but I wanted to through this in the mix.

The first example (http://ask.github.com/celery/getting-started/introduction.html) says it all. You can fire tasks from anywhere, such as in a view, or as a signal receiver, etc. Periodic tasks (see http://ask.github.com/celery/userguide/periodic-tasks.html) offer scheduled jobs. I use both.

I think that the django-celery-email (http://pypi.python.org/pypi/django-celery-email / https://github.com/pmclanahan/django-celery-email) project is a great starting point. Take a look at the code for a simple, yet powerful, example which isn't much more complicated than the celery docs themselves.

I had the most trouble getting the task results to work in the admin, then I found out about celerycam (http://ask.github.com/celery/userguide/monitoring.html#starting-the-monitor), which I manage with supervisor.

-Ryan

On Thu, Feb 2, 2012 at 9:36 AM, Tung Wai Yip <[hidden email]> wrote:
I'm using RabbitMQ and Celery. It certainly can be triggered by both cases. In addition, you can deploy it on different servers if necessary.

Wai Yip


I use Django in production and now have a need to perform asynchronous
backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and
celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is
steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?

I am looking for some opinions /recommendations to use one of  these two
technologies or something else altogether ?
Thanks
Meenal
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Alec Flett-2
In reply to this post by Meenal Pant-2
A little late to the party but...

On Thu, Feb 2, 2012 at 8:28 AM, Meenal Pant <[hidden email]> wrote:
I use Django in production and now have a need to perform asynchronous backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?


I actually did this with Twisted inside of Pylons a few years ago - this was before all these queuing systems were popular. 

While everything actually did work great, I can describe drawbacks of using Twisted for this specific case right off the bat:

1) the architecture of Twisted has very tight bindings to a single-threaded process model - meaning that everything wants to use the global reactor. This means if you have a multi-threaded server, you have to keep thread-local reactors around and monkeypatch bits of Twisted to get it to behave
2) It's a little complicated to keep track of all the outstanding deferreds to make sure you return your response after all of them complete - basically you want a "wait for all deferred to resolve" but there isn't a single global version of that for twisted - there are various permutations of inspecting the reactor, but ultimately you have to do the accounting yourself.. 

Overall I wish there was a system that matched the existing request/thread model that I was using (Pylons) rather than bolting on a secondary system. At the time there weren't any other options, but I'm sure now there are a bunch of better alternatives

Alec



I am looking for some opinions /recommendations to use one of  these two technologies or something else altogether ?
Thanks
Meenal

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Shannon -jj Behrens
Alec, if I had to do what you did at Metaweb again today, I'd probably use gevent, and I'd run Pylons on top of gevent (aka gunicorn).  However, I think your use case was a little bit different than a standard queuing problem.  You were trying to execute multiple queries in parallel within a single request, whereas most people who use queues are using them to queue up requests to be done after the web request is done.

I still think your Twisted magic at Metaweb was amazing :)

Best Regards,
-jj

On Mon, Feb 6, 2012 at 10:31 AM, Alec Flett <[hidden email]> wrote:
A little late to the party but...

On Thu, Feb 2, 2012 at 8:28 AM, Meenal Pant <[hidden email]> wrote:
I use Django in production and now have a need to perform asynchronous backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?


I actually did this with Twisted inside of Pylons a few years ago - this was before all these queuing systems were popular. 

While everything actually did work great, I can describe drawbacks of using Twisted for this specific case right off the bat:

1) the architecture of Twisted has very tight bindings to a single-threaded process model - meaning that everything wants to use the global reactor. This means if you have a multi-threaded server, you have to keep thread-local reactors around and monkeypatch bits of Twisted to get it to behave
2) It's a little complicated to keep track of all the outstanding deferreds to make sure you return your response after all of them complete - basically you want a "wait for all deferred to resolve" but there isn't a single global version of that for twisted - there are various permutations of inspecting the reactor, but ultimately you have to do the accounting yourself.. 

Overall I wish there was a system that matched the existing request/thread model that I was using (Pylons) rather than bolting on a secondary system. At the time there weren't any other options, but I'm sure now there are a bunch of better alternatives

Alec



I am looking for some opinions /recommendations to use one of  these two technologies or something else altogether ?
Thanks
Meenal

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies



--
In this life we cannot do great things. We can only do small things with great love. -- Mother Teresa

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Meenal Pant-2
 Thanks everyone! I think I will go with celery and RabbitMQ.
Meenal

On Mon, Feb 6, 2012 at 4:29 PM, Shannon -jj Behrens <[hidden email]> wrote:
Alec, if I had to do what you did at Metaweb again today, I'd probably use gevent, and I'd run Pylons on top of gevent (aka gunicorn).  However, I think your use case was a little bit different than a standard queuing problem.  You were trying to execute multiple queries in parallel within a single request, whereas most people who use queues are using them to queue up requests to be done after the web request is done.

I still think your Twisted magic at Metaweb was amazing :)

Best Regards,
-jj

On Mon, Feb 6, 2012 at 10:31 AM, Alec Flett <[hidden email]> wrote:
A little late to the party but...

On Thu, Feb 2, 2012 at 8:28 AM, Meenal Pant <[hidden email]> wrote:
I use Django in production and now have a need to perform asynchronous backend tasks that
a. Are some scheduled jobs
b. Jobs triggered via a user input via Django frontend.

I have been thinking of using twisted or message queues (RabbitMQ and celery) to achieve this.  From my research so far:

1. Twisted allows both a and b though the learning curve with twisted is steeper
2. RabbitMQ and celery is easier to use, allows a but does it also allow b?


I actually did this with Twisted inside of Pylons a few years ago - this was before all these queuing systems were popular. 

While everything actually did work great, I can describe drawbacks of using Twisted for this specific case right off the bat:

1) the architecture of Twisted has very tight bindings to a single-threaded process model - meaning that everything wants to use the global reactor. This means if you have a multi-threaded server, you have to keep thread-local reactors around and monkeypatch bits of Twisted to get it to behave
2) It's a little complicated to keep track of all the outstanding deferreds to make sure you return your response after all of them complete - basically you want a "wait for all deferred to resolve" but there isn't a single global version of that for twisted - there are various permutations of inspecting the reactor, but ultimately you have to do the accounting yourself.. 

Overall I wish there was a system that matched the existing request/thread model that I was using (Pylons) rather than bolting on a secondary system. At the time there weren't any other options, but I'm sure now there are a bunch of better alternatives

Alec



I am looking for some opinions /recommendations to use one of  these two technologies or something else altogether ?
Thanks
Meenal

_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies



--
In this life we cannot do great things. We can only do small things with great love. -- Mother Teresa


_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Aahz
In reply to this post by Tung Wai Yip
Digressing a bit...

On Thu, Feb 02, 2012, Tung Wai Yip wrote:
>
> I'm using RabbitMQ and Celery. It certainly can be triggered by both
> cases. In addition, you can deploy it on different servers if
> necessary.

Do people really find that RabbitMQ is stable enough for production work?
What kinds of volumes?  (I wasn't involved with it myself, but about 1.5
years ago we couldn't get RabbitMQ stable enough for our needs --
millions of messages per day -- and ended up switching to Scribe.)
--
Aahz ([hidden email])           <*>         http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies
Reply | Threaded
Open this post in threaded view
|

Re: Asynchronous web tasks

Michael Pittaro-3
On Wed, Feb 8, 2012 at 7:43 AM, Aahz <[hidden email]> wrote:

> Digressing a bit...
>
> On Thu, Feb 02, 2012, Tung Wai Yip wrote:
>>
>> I'm using RabbitMQ and Celery. It certainly can be triggered by both
>> cases. In addition, you can deploy it on different servers if
>> necessary.
>
> Do people really find that RabbitMQ is stable enough for production work?
> What kinds of volumes?  (I wasn't involved with it myself, but about 1.5
> years ago we couldn't get RabbitMQ stable enough for our needs --
> millions of messages per day -- and ended up switching to Scribe.)

OpenStack uses Rabbit as it's core messaging system (Carrot as the
client side.)

It's in production is quite a few locations - not perfect but
definitely stable enough for production work.

mike
_______________________________________________
Baypiggies mailing list
[hidden email]
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies