multi threaded/process web apps

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

multi threaded/process web apps

James Alford
Hi All

I have reached an impasse with a php based web services project.
Doing multi threaded/process operations is just not a simple affiar.

Can any one recommend a python web framework that seamlessly does
multi threaded/process operations?

I'm reviewing webpy and twisted today as possible replacements.

Thanks
James
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Andreux Fort-2

Webpy is agnostic, it's not a threaded server.  Twisted is a single threaded approach to the problem, but is also a framework.  Django, webpy, pylons etc will all work with multithreaded servers.

For an asynchronous approach try twisted or tornado.

On Jul 10, 2011 7:27 PM, "James Alford" <[hidden email]> wrote:
> Hi All
>
> I have reached an impasse with a php based web services project.
> Doing multi threaded/process operations is just not a simple affiar.
>
> Can any one recommend a python web framework that seamlessly does
> multi threaded/process operations?
>
> I'm reviewing webpy and twisted today as possible replacements.
>
> Thanks
> James
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Anthony Briggs-3
In reply to this post by James Alford
Hi James,

More details would probably help (eg. what's the project/what are you
trying to do). There might be an alternative way to do things.

In general though, "multi-threaded" and "seamless" don't go together
very well. Even in frameworks like Twisted, there's a significant
overhead of deferreds, callbacks, errbacks, etc. which will make your
code hard to follow. Easier than doing it in standard Python, but
still hard.

Anthony

On 11 July 2011 12:27, James Alford <[hidden email]> wrote:

> Hi All
>
> I have reached an impasse with a php based web services project.
> Doing multi threaded/process operations is just not a simple affiar.
>
> Can any one recommend a python web framework that seamlessly does
> multi threaded/process operations?
>
> I'm reviewing webpy and twisted today as possible replacements.
>
> Thanks
> James
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Rory Hart-3
What are your requirements? 

It sounds like what you might be wanting to do is do some processing that is currently tying up your web server for long periods of time.

If that is the case I would suggest any of the big python web frameworks and Celery (http://celeryproject.org/) which is a task queue and has integrations with many of the frameworks.

Rory

On Mon, Jul 11, 2011 at 12:33 PM, Anthony Briggs <[hidden email]> wrote:
Hi James,

More details would probably help (eg. what's the project/what are you
trying to do). There might be an alternative way to do things.

In general though, "multi-threaded" and "seamless" don't go together
very well. Even in frameworks like Twisted, there's a significant
overhead of deferreds, callbacks, errbacks, etc. which will make your
code hard to follow. Easier than doing it in standard Python, but
still hard.

Anthony

On 11 July 2011 12:27, James Alford <[hidden email]> wrote:
> Hi All
>
> I have reached an impasse with a php based web services project.
> Doing multi threaded/process operations is just not a simple affiar.
>
> Can any one recommend a python web framework that seamlessly does
> multi threaded/process operations?
>
> I'm reviewing webpy and twisted today as possible replacements.
>
> Thanks
> James
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

James Alford
In reply to this post by Anthony Briggs-3
Ok, that was quick and rightfully flamed for not being specific.

This is the scenario I have solved in the past on the ipad but with
php it is a litte complex.

Scenario:
1. Incoming web service request
2. Data is processed
3. Outgoing web request sent.
4. Web service request response sent

With step 3, I would fired an event or spawn another thread to do the
outgoing request.  With php it is possible but I have to start a
separate process as a php cli process.  With the ipad I'm easily able
to fire an event that handles this in a separate thread.

Any help would be appreciated.

On Mon, Jul 11, 2011 at 12:33 PM, Anthony Briggs
<[hidden email]> wrote:

> Hi James,
>
> More details would probably help (eg. what's the project/what are you
> trying to do). There might be an alternative way to do things.
>
> In general though, "multi-threaded" and "seamless" don't go together
> very well. Even in frameworks like Twisted, there's a significant
> overhead of deferreds, callbacks, errbacks, etc. which will make your
> code hard to follow. Easier than doing it in standard Python, but
> still hard.
>
> Anthony
>
> On 11 July 2011 12:27, James Alford <[hidden email]> wrote:
>> Hi All
>>
>> I have reached an impasse with a php based web services project.
>> Doing multi threaded/process operations is just not a simple affiar.
>>
>> Can any one recommend a python web framework that seamlessly does
>> multi threaded/process operations?
>>
>> I'm reviewing webpy and twisted today as possible replacements.
>>
>> Thanks
>> James
>> _______________________________________________
>> melbourne-pug mailing list
>> [hidden email]
>> http://mail.python.org/mailman/listinfo/melbourne-pug
>>
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Rory Hart-3
Right well then the solution I described would work for you.

Some examples:

Also I am all for killing off PHP code[1] but if you have a lot of code to move there are fully fledged PHP options.


Rory

[1] Currently replacing some PHP code with heavily nested for loops that takes 30 minutes and many hundreds of megs to run with a couple of SQL queries that take a couple of seconds in total. Though to be fair to PHP you could create such a monstrosity in Python just as easily, but Python programmers as a group appear to far more adept programmers.

On Mon, Jul 11, 2011 at 12:58 PM, James Alford <[hidden email]> wrote:
Ok, that was quick and rightfully flamed for not being specific.

This is the scenario I have solved in the past on the ipad but with
php it is a litte complex.

Scenario:
1. Incoming web service request
2. Data is processed
3. Outgoing web request sent.
4. Web service request response sent

With step 3, I would fired an event or spawn another thread to do the
outgoing request.  With php it is possible but I have to start a
separate process as a php cli process.  With the ipad I'm easily able
to fire an event that handles this in a separate thread.

Any help would be appreciated.

On Mon, Jul 11, 2011 at 12:33 PM, Anthony Briggs
<[hidden email]> wrote:
> Hi James,
>
> More details would probably help (eg. what's the project/what are you
> trying to do). There might be an alternative way to do things.
>
> In general though, "multi-threaded" and "seamless" don't go together
> very well. Even in frameworks like Twisted, there's a significant
> overhead of deferreds, callbacks, errbacks, etc. which will make your
> code hard to follow. Easier than doing it in standard Python, but
> still hard.
>
> Anthony
>
> On 11 July 2011 12:27, James Alford <[hidden email]> wrote:
>> Hi All
>>
>> I have reached an impasse with a php based web services project.
>> Doing multi threaded/process operations is just not a simple affiar.
>>
>> Can any one recommend a python web framework that seamlessly does
>> multi threaded/process operations?
>>
>> I'm reviewing webpy and twisted today as possible replacements.
>>
>> Thanks
>> James
>> _______________________________________________
>> melbourne-pug mailing list
>> [hidden email]
>> http://mail.python.org/mailman/listinfo/melbourne-pug
>>
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug



--
Rory Hart
http://www.roryhart.net

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

William Leslie-2
In reply to this post by James Alford
On 11 July 2011 12:58, James Alford <[hidden email]> wrote:

> Ok, that was quick and rightfully flamed for not being specific.
>
> This is the scenario I have solved in the past on the ipad but with
> php it is a litte complex.
>
> Scenario:
> 1. Incoming web service request
> 2. Data is processed
> 3. Outgoing web request sent.
> 4. Web service request response sent
>
> With step 3, I would fired an event or spawn another thread to do the
> outgoing request.  With php it is possible but I have to start a
> separate process as a php cli process.  With the ipad I'm easily able
> to fire an event that handles this in a separate thread.

This list is more or less your twisted callback chain.  The nature of
"Data is processed" determines exactly how you make it scale with
twisted - for example, if it is a bunch of queries you want to run,
you could use the adbapi on top of your existing database driver.  If
you want to interact with a number of internal services, maybe AMP is
the way to go.  Otherwise, you could start a bunch of subprocesses.

Starting a separate thread per request scales (in performance) poorly,
and that is even more true on cpython, so it should be avoided where
possible.  Starting separate processes or interacting asynchronously
with the web server tends to be faster besides resulting in code that
is easier to comprehend.

--
William Leslie
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

James Alford
Hi All

Thanks for the responses.

@Rory - I had actually forgotten about django, thanks.

@William - I will definitely look at processes.

It has given me something to think about.  Django and webpy look like
the way to go but I will also have a look at twisted.

Thanks again.

James
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

paul sorenson-6
On 11/07/11 16:02, James Alford wrote:

> Hi All
>
> Thanks for the responses.
>
> @Rory - I had actually forgotten about django, thanks.
>
> @William - I will definitely look at processes.
>
> It has given me something to think about.  Django and webpy look like
> the way to go but I will also have a look at twisted.
>
> Thanks again.
>
> James
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
cherrypy (http://www.cherrypy.org/) is multi threaded.  I haven't used
it on anything but a proof of concept scale and it did what I asked of it.
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Rasjid Wilcox
On 11/07/11 19:30, paul sorenson wrote:
> cherrypy (http://www.cherrypy.org/) is multi threaded.  I haven't used
> it on anything but a proof of concept scale and it did what I asked of
> it.

I have used CherryPy in production, and have been very happy with it.  
Unlike Django, it comes with its own production ready web server written
in Python, so it is trivial to deploy.  (On linux I run it under
Supervisor - http://supervisord.org/ - also written in Python.  On
Windows it can be run as a Service.)  IMO it is more 'pythonic' than
Django, and allows you to use whatever ORM you want (or none if none are
required).

Other lightweight frameworks that I've been I have looked at briefly but
not used in production are:

Flask (http://flask.pocoo.org/)
Bottle (http://bottlepy.org/docs/dev/)
Aspen (http://aspen.io/)

Aspen is really interesting in that it is based around the filesystem
rather than routes etc, and is much closer to PHP is style and
simplicity, but with Python goodness.

Less lightweight but worth looking at:

Pyramid (http://docs.pylonsproject.org/docs/pyramid.html) which is built
on Pylons.

For something completely different, there is Nagare
(http://www.nagare.org/) that runs on top of Stackless Python, and looks
really interesting.  Not sure if it would be a good fit in this case,
but looks like some very cool tech.

Cheers,

Rasjid.

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Richard Jones-7
PLUG TIME! :-)

People interested in light-weight web frameworks might be interested
in my talk coming up at PyCon AU "Web Micro-Framework Battle."

So far I've looked into: aspen.io, bobo, bottle, celery, cgi+basehttp,
cherrypy, flask, grok, itty, milla, pesto, pyramid, routes+webob,
web.py, web2py and werkzeug.

Suggestions for additional frameworks to look at are welcome.


     Richard
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Rasjid Wilcox
Hi Richard,

Do you know if the talks be recorded and made available online?  Your
talk covers all the frameworks that I'm interested in (plus several
others I've not head of), sans Nagare as per my prior post.  (Not sure
if Nagare counts as a micro-framework or not - so may or may not fit in
with your talk.)  Unfortunately I'm unable to make it to PyCon due to
family commitments, but would still love to see you talk.

Cheers,

Rasjid.


On 12/07/11 11:02, Richard Jones wrote:

> PLUG TIME! :-)
>
> People interested in light-weight web frameworks might be interested
> in my talk coming up at PyCon AU "Web Micro-Framework Battle."
>
> So far I've looked into: aspen.io, bobo, bottle, celery, cgi+basehttp,
> cherrypy, flask, grok, itty, milla, pesto, pyramid, routes+webob,
> web.py, web2py and werkzeug.
>
> Suggestions for additional frameworks to look at are welcome.
>
>
>       Richard
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Richard Jones-7
On 14 July 2011 10:07, Rasjid Wilcox <[hidden email]> wrote:
> Do you know if the talks be recorded and made available online?

Yes, but it's not the same as being there in person :-)


     Richard
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Rasjid Wilcox
On 14/07/11 10:12, Richard Jones wrote:
> On 14 July 2011 10:07, Rasjid Wilcox<[hidden email]>  wrote:
>> Do you know if the talks be recorded and made available online?
> Yes, but it's not the same as being there in person :-)
>
Absolutely not!  I'd love to be there, but that weekend my elder son is
coming down to visit, and since I only see him a couple of times a year,
that wins.  :-)

Cheers,

Rasjid.

_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

Graeme Cross
On Thu, 14 Jul 2011 10:29 +1000, "Rasjid Wilcox"
<[hidden email]> wrote:

> On 14/07/11 10:12, Richard Jones wrote:
> > On 14 July 2011 10:07, Rasjid Wilcox<[hidden email]>  wrote:
> >> Do you know if the talks be recorded and made available online?
> > Yes, but it's not the same as being there in person :-)
> >
> Absolutely not!  I'd love to be there, but that weekend my elder son is
> coming down to visit, and since I only see him a couple of times a year,
> that wins.  :-)
>
> Cheers,
> Rasjid.
>

Rasjid, you could always bring your son to PyCon AU ;)

Best of both worlds then!

- Graeme
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug
Reply | Threaded
Open this post in threaded view
|

Re: multi threaded/process web apps

James Alford
In reply to this post by Rasjid Wilcox
After a bit of research and implementation trials I have settled somewhat.

For the moment rabbitmq and python with pika (rabbitmq client) will
plug some of the holes with the php project.  Once that is stable I
will move the php project to maintenance releases and start
redevelopment in django.

Moving forward i'm looking at a django, celery and rabiitmq combo to
give me what I need.  PHP is great for relatively simple web service
based web apps.  That is probably where it should stay.

Thanks to everyone for your input and advice.

On Tue, Jul 12, 2011 at 10:44 AM, Rasjid Wilcox <[hidden email]> wrote:

> On 11/07/11 19:30, paul sorenson wrote:
>>
>> cherrypy (http://www.cherrypy.org/) is multi threaded.  I haven't used it
>> on anything but a proof of concept scale and it did what I asked of it.
>
> I have used CherryPy in production, and have been very happy with it.
>  Unlike Django, it comes with its own production ready web server written in
> Python, so it is trivial to deploy.  (On linux I run it under Supervisor -
> http://supervisord.org/ - also written in Python.  On Windows it can be run
> as a Service.)  IMO it is more 'pythonic' than Django, and allows you to use
> whatever ORM you want (or none if none are required).
>
> Other lightweight frameworks that I've been I have looked at briefly but not
> used in production are:
>
> Flask (http://flask.pocoo.org/)
> Bottle (http://bottlepy.org/docs/dev/)
> Aspen (http://aspen.io/)
>
> Aspen is really interesting in that it is based around the filesystem rather
> than routes etc, and is much closer to PHP is style and simplicity, but with
> Python goodness.
>
> Less lightweight but worth looking at:
>
> Pyramid (http://docs.pylonsproject.org/docs/pyramid.html) which is built on
> Pylons.
>
> For something completely different, there is Nagare (http://www.nagare.org/)
> that runs on top of Stackless Python, and looks really interesting.  Not
> sure if it would be a good fit in this case, but looks like some very cool
> tech.
>
> Cheers,
>
> Rasjid.
>
> _______________________________________________
> melbourne-pug mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/melbourne-pug
>
_______________________________________________
melbourne-pug mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/melbourne-pug