Alternative to celery

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Alternative to celery

guettli-3
Some years ago I used celery for a project.

Today I face roughly the same use case again.

The last time I used it, celery felt to heavy weighted, too complicated.

I found this alternative: http://python-rq.org/

There are two things which speak agains python-rq:

 - redis is a cache. I need a persistent queue. If a power-failure happens, no job must get lost.

 - AFAIK python-rq uses python pickle to serialize data. I want a language neutral data exchange format.

At the moment my favourite protocol is gRPC (protocol-buffers). I never used it up to now, but this would be a nice use case.

Maybe I am missing the right terms. What does celery implement? According to wikipedia "asynchronous task queue". But
my favorite search engine could not reveal a gRPC based server implemented in Python ....

What do you think?


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Alternative to celery

Andréas Kühne
I don't know anything about the python-rq stuff - however redis can be configured to save to disk (see https://redis.io/topics/persistence) - so you wouldn't loose anything on a power-failure (if configured correctly). Redis is so much more than a cache nowadays.

Regards,

Andréas

2017-04-20 11:16 GMT+02:00 guettli <[hidden email]>:
Some years ago I used celery for a project.

Today I face roughly the same use case again.

The last time I used it, celery felt to heavy weighted, too complicated.

I found this alternative: http://python-rq.org/

There are two things which speak agains python-rq:

 - redis is a cache. I need a persistent queue. If a power-failure happens, no job must get lost.

 - AFAIK python-rq uses python pickle to serialize data. I want a language neutral data exchange format.

At the moment my favourite protocol is gRPC (protocol-buffers). I never used it up to now, but this would be a nice use case.

Maybe I am missing the right terms. What does celery implement? According to wikipedia "asynchronous task queue". But
my favorite search engine could not reveal a gRPC based server implemented in Python ....

What do you think?


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCefVFpgZEw4Ar4AGDUSdOH-A-7-kvgysqeHY9zET1DHZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Alternative to celery

Avraham Serour
try googling 'python task queue'


On Thu, Apr 20, 2017 at 12:25 PM, Andréas Kühne <[hidden email]> wrote:
I don't know anything about the python-rq stuff - however redis can be configured to save to disk (see https://redis.io/topics/persistence) - so you wouldn't loose anything on a power-failure (if configured correctly). Redis is so much more than a cache nowadays.

Regards,

Andréas

2017-04-20 11:16 GMT+02:00 guettli <[hidden email]>:
Some years ago I used celery for a project.

Today I face roughly the same use case again.

The last time I used it, celery felt to heavy weighted, too complicated.

I found this alternative: http://python-rq.org/

There are two things which speak agains python-rq:

 - redis is a cache. I need a persistent queue. If a power-failure happens, no job must get lost.

 - AFAIK python-rq uses python pickle to serialize data. I want a language neutral data exchange format.

At the moment my favourite protocol is gRPC (protocol-buffers). I never used it up to now, but this would be a nice use case.

Maybe I am missing the right terms. What does celery implement? According to wikipedia "asynchronous task queue". But
my favorite search engine could not reveal a gRPC based server implemented in Python ....

What do you think?


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCefVFpgZEw4Ar4AGDUSdOH-A-7-kvgysqeHY9zET1DHZQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFWa6t%2B7WTzt7o1SoOV45YrL%2B%3DaD_X30E1yNFj3-ki54ivNL2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Alternative to celery

Marcin Nowak
In reply to this post by guettli-3
RQ is simple, stable and reliable. Redis is also a persistent storage, but you should read about this and use the best solution for your case: https://redis.io/topics/persistence   
You may also consider deploying a fault-tolerant Redis cluster (min. 3 nodes AFAIR)

RQ has different architecture than Celery. It is based on fork(), which is just slower. So you should ask yourself how many tasks per sec you need to handle. RQ may not be enough for all use cases.
Some things aren't OOTB, like repeating failed tasks (you should handle it manually, i.e. by using own worker class), nor  tasks scheduling (you should use rq-scheduler). 

RQ uses pickle, but there is a feature request about handling json and other serializers - https://github.com/nvie/rq/issues/369 


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d2fc1e0f-f147-4ccd-991f-1b12e61c8964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...