threadsafe patch for asynchat

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

threadsafe patch for asynchat

Mark Edgington
Does anyone have any comments about applying the following patch to
asynchat?  It should not affect the behavior of the module in any way
for those who do not want to use the feature provided by the patch.  The
point of the patch is to make it easy to use asynchat in a multithreaded
application.  Maybe I am missing something, and the patch really doesn't
make it threadsafe?  Any comments would be appreciated.  Also, if it
looks good to everyone, feel free to use it.

-------BEGIN PATCH----------
--- asynchat.py    Fri Oct 15 03:03:16 2004
+++ asynchat.py.new    Sun Feb 05 22:05:42 2006
@@ -59,10 +59,11 @@
     ac_in_buffer_size       = 4096
     ac_out_buffer_size      = 4096
 
-    def __init__ (self, conn=None):
+    def __init__ (self, conn=None, running_in_thread=False):
         self.ac_in_buffer = ''
         self.ac_out_buffer = ''
         self.producer_fifo = fifo()
+        self.running_in_thread = runnning_in_thread
         asyncore.dispatcher.__init__ (self, conn)
 
     def collect_incoming_data(self, data):
@@ -157,7 +158,9 @@
 
     def push (self, data):
         self.producer_fifo.push (simple_producer (data))
-        self.initiate_send()
+        # only initiate a send if not running in a threaded
environment, since
+        # initiate_send() is not threadsafe.
+        if not self.running_in_thread: self.initiate_send()
 
     def push_with_producer (self, producer):
         self.producer_fifo.push (producer)
-------END PATCH----------

-Mark

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

"Martin v. Löwis"
Mark Edgington wrote:
> Does anyone have any comments about applying the following patch to
> asynchat?

That patch looks wrong. What does it mean to "run in a thread"?
All code runs in a thread, all the time: sometime, that thread
is the main thread.

Furthermore, I can't see any presumed thread-unsafety in asynchat.

Sure, there is a lot of member variables in asynchat which aren't
specifically protected against mutual access from different threads.
So you shouldn't be accessing the same async_chat object from multiple
threads. I cannot see why using a creating and using
an async_chat object in a thread that is not the main thread
could cause any problems. I also cannot see how this patch could
have significant effect on asyn_chat's behaviour when used in
multiple threads.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Mark Edgington
In reply to this post by Mark Edgington
Martin v. Löwis wrote:
 > That patch looks wrong. What does it mean to "run in a thread"?
 > All code runs in a thread, all the time: sometime, that thread
 > is the main thread.
 >
 > Furthermore, I can't see any presumed thread-unsafety in asynchat.

Ok, perhaps the notation could be improved, but the idea of the
semaphore in the patch is "Does it run inside of a multithreaded
environment, and could its push() functions be called from a different
thread?"

I have verified that there is a problem with running it in such an
environment.  My results are more or less identical to those described
in the following thread:
http://mail.python.org/pipermail/medusa-dev/1999/000431.html
(see also the reply message to this one regarding the solution -- if you
look at the Zope source, Zope deals with the problem in the way I am
suggesting asynchat be patched)

It seems that somehow in line 271 (python 2.4) of asynchat.py,
producer_fifo.list is not empty, and thus popleft() is executed.
However, popleft() finds the deque empty.  This means that somehow the
deque (or list -- the bug is identical in python 2.3) is emptied between
the if() and the popleft(), so perhaps asyncore.loop(), running in a
different thread from the thread which calls async_chat.push(), empties it.

The problem is typically exhibited when running in a multithreaded
environment, and when calling the async_chat.push() function many (i.e.
perhaps tens of thousands) times quickly in a row from a different
thread.  However, this behavior is avoided by creating a Lock for
refill_buffer(), so that it cannot be executed simultaneously.  It is
also avoided by not executing initiate_send() at all (as is done by Zope
in ZHTTPServer.zhttp_channel).

 > Sure, there is a lot of member variables in asynchat which aren't
 > specifically protected against mutual access from different threads.
 > So you shouldn't be accessing the same async_chat object from multiple
 > threads.

If applying this patch does indeed make it safe to use async_chat.push()
from other threads, why would it be a bad thing to have?  It seems to
make the code less cryptic (i.e. I don't need to override base classes
in order to include code which processes a nonempty Queue object -- I
simply make a call to the push() function of my instance of async_chat,
and I'm done).

-Mark

(also, of course push_with_producer() would probably also need the same
changes that would be made to push() )

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Guido van Rossum
In reply to this post by "Martin v. Löwis"
IMO asynchat and asyncore are braindead. The should really be removed
from the standard library. The code is 10 years old and represents at
least 10-year-old thinking about how to do this. The amount of hackery
in Zope related to asyncore was outrageous -- basically most of
asyncore's guts were replaced with more advanced Zope code, but the
API was maintained for compatibility reasons. A nightmare.

--Guido

On 2/6/06, "Martin v. Löwis" <[hidden email]> wrote:

> Mark Edgington wrote:
> > Does anyone have any comments about applying the following patch to
> > asynchat?
>
> That patch looks wrong. What does it mean to "run in a thread"?
> All code runs in a thread, all the time: sometime, that thread
> is the main thread.
>
> Furthermore, I can't see any presumed thread-unsafety in asynchat.
>
> Sure, there is a lot of member variables in asynchat which aren't
> specifically protected against mutual access from different threads.
> So you shouldn't be accessing the same async_chat object from multiple
> threads. I cannot see why using a creating and using
> an async_chat object in a thread that is not the main thread
> could cause any problems. I also cannot see how this patch could
> have significant effect on asyn_chat's behaviour when used in
> multiple threads.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Fredrik Lundh
Guido van Rossum wrote:

> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this.

strange.  I'd say it works perfectly fine for what it was designed for
(after all, sockets haven't changed much in 10 years either).

what other reactive socket framework is there that would fit well into
the standard library ?  is twisted really simple enough ?

</F>



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Robert Brewer
In reply to this post by Mark Edgington
Guido van Rossum wrote:
> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this. The amount of hackery
> in Zope related to asyncore was outrageous -- basically most of
> asyncore's guts were replaced with more advanced Zope code, but the
> API was maintained for compatibility reasons. A nightmare.

Perhaps, but please keep in mind that the smtpd module uses both, currently, and would have to be rewritten if either is "removed".


Robert Brewer
System Architect
Amor Ministries
[hidden email]
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Alex Martelli-4
In reply to this post by Fredrik Lundh
On 2/7/06, Fredrik Lundh <[hidden email]> wrote:
   ...
> what other reactive socket framework is there that would fit well into
> the standard library ?  is twisted really simple enough ?

Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
subset could be carefully extracted that (given suitable volunteers to
maintain it in the future) might fit in the standard library, but [a]
that extraction is not going to be a simple or fast job, and [b] I
suspect that the minimum sensible subset would still be much larger
(and richer / more powerful) than asyncore.


Alex
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Josiah Carlson
In reply to this post by Guido van Rossum

Guido van Rossum <[hidden email]> wrote:
> IMO asynchat and asyncore are braindead. The should really be removed
> from the standard library. The code is 10 years old and represents at
> least 10-year-old thinking about how to do this. The amount of hackery
> in Zope related to asyncore was outrageous -- basically most of
> asyncore's guts were replaced with more advanced Zope code, but the
> API was maintained for compatibility reasons. A nightmare.

I'm going to go ahead and disagree with Guido on this one.  Before
removing asyncore (and asynchat) from the standard library, I believe
that there would necessarily need to be a viable replacement already
in place. The SocketServer module and its derivatives are wholly
unscalable for server-oriented applications once you get past a few
dozen threads (where properly designed asyncore derivatives will do
quite well all the way to your platform file handle limit).

Every once and a while I hear about people pushing for Twisted to be
included with Python, but at 2 megs for the base bz2 package, it seems a
little...hefty.  I'm not aware of any other community-accepted package
for asynchronous socket clients and servers, but I'm always looking.

Now, don't get me wrong, writing servers and clients using asyncore or
asynchat can be a beast, but it does get one into the callback/reactor
method of programming, which seems to have invaded other parts of Python
and 3rd party libraries (xml.sax, tk, Twisted, wxPython, ...).

Back to the topic that Guido was really complaining about: Zope +
asyncore.  I don't doubt that getting Zope to play nicely with asyncore
was difficult, but it begs the questions: what would have been done if
asyncore didn't exist, and why wasn't that done instead of trying to
play nicely with asyncore?

 - Josiah

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Barry Warsaw
In reply to this post by Robert Brewer
On Tue, 2006-02-07 at 16:01 -0800, Robert Brewer wrote:

> Perhaps, but please keep in mind that the smtpd module uses both, currently, and would have to be rewritten if either is "removed".

Would that really be a huge loss?

-Barry


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Christopher Armstrong-2
In reply to this post by Alex Martelli-4
On 2/8/06, Alex Martelli <[hidden email]> wrote:

> On 2/7/06, Fredrik Lundh <[hidden email]> wrote:
>    ...
> > what other reactive socket framework is there that would fit well into
> > the standard library ?  is twisted really simple enough ?
>
> Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
> subset could be carefully extracted that (given suitable volunteers to
> maintain it in the future) might fit in the standard library, but [a]
> that extraction is not going to be a simple or fast job, and [b] I
> suspect that the minimum sensible subset would still be much larger
> (and richer / more powerful) than asyncore.

The subject of putting (parts of) Twisted into the standard library
comes up once every 6 months or so, at least on our mailing list. For
all that I think asyncore is worthless, I'm still against copying
Twisted into the stdlib. Or at least I'm not willing to maintain the
necessary fork, and I fear the nightmares about versioning that can
easily occur when you've got both standard library and third party
versions of a project.

But, for the record, to the people who argue not to put Twisted into
the stdlib because of its size: The parts of it that would actually be
applicable (i.e. those that obselete async* in the stdlib) are only a
few kilobytes of code. At a quick run of "wc", the parts that support
event loops, accurate timed calls, SSL, Unix sockets, TCP, UDP,
arbitrary file descriptors, processes, and threads sums up to about
5300 lines of code. asynchat and asyncore are about 1200.

--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix    |    -- http://radix.twistedmatrix.com
            |  Release Manager, Twisted Project
  \\\V///   |    -- http://twistedmatrix.com
   |o O|    |
w----v----w-+
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Bugzilla from nnorwitz@gmail.com
On 2/7/06, Christopher Armstrong <[hidden email]> wrote:

>
> > Twisted is wonderful, powerful, rich, and very large.  Perhaps a small
> > subset could be carefully extracted
>
> The subject of putting (parts of) Twisted into the standard library
> comes up once every 6 months or so, at least on our mailing list. For
> all that I think asyncore is worthless, I'm still against copying
> Twisted into the stdlib. Or at least I'm not willing to maintain the
> necessary fork, and I fear the nightmares about versioning that can
> easily occur when you've got both standard library and third party
> versions of a project.

I wouldn't be enthusiastic about putting all of Twisted in the stdlib
either.  Twisted is on a different release schedule than Python.
However, isn't there a relatively small core subset like Alex
mentioned that isn't changing much?  Could we split up those
components and have those live in the core, but the vast majority of
Twisted live outside as it does now?

n
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

jalopyuser
In reply to this post by Fredrik Lundh
> what other reactive socket framework is there that would fit well into
> the standard library ?  is twisted really simple enough ?

I've been very happy with Medusa, which is asyncore-based.

Perhaps the right idea is to fix the various problems of asyncore.  We
might lift the similar code from the kernel of ILU, for example, which
carefully addresses the various issues around this style of action loop.

Bill
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Tim Peters
In reply to this post by Josiah Carlson
[Josiah Carlson]
> ...
> Back to the topic that Guido was really complaining about: Zope +
> asyncore.  I don't doubt that getting Zope to play nicely with asyncore
> was difficult,

It's more that mixing asyncore with threads is a bloody nightmare, and
ZEO and Zope both do that.  Zope (but not ZEO) goes on to mix threads
with asynchat too.  In addition, ZEO makes life much harder than
should be necessary by running in two different modes and
auto-switching between them, depending on whether "the app" is or is
not running an asyncore mainloop itself.  In order to _detect_ when
"the app" fires up an asyncore mainloop, ZEO monkey-patches asyncore's
loop() function and physically replaces it with its own loop()
function.  It goes downhill from there.

Guido's memories are partly out of date now:  ZEO used to replace a
lot more of asyncore than it does now, because of bugs in the asyncore
distributed with older Python versions.  The _needs_ for that went
away little by little over the years, but the code in ZEO stuck around
much longer.  ZEO's ThreadedAsync/LoopCallback.py is much smaller now
(ZODB 3.6) than Guido remembers.

For a brief while, I even ripped out ZEO's monkey-patching of Python's
asyncore loop(), but it turned out that newer code in Zope3 (but not
Zope2) relied on, in turn, poking values into ZEO's module globals to
cause ZEO's loop() replacement to shut down (that's the kind of
"expedient" joy you get when mixing asyncore with threads).

Every piece of it remains "underdocumented" and, IMO, highly obscure.

> but it begs the questions: what would have been done if asyncore didn't exist,

Who knows?  What would python-dev be like if you didn't exist :-)?

> and why wasn't that done instead of trying to play nicely with asyncore?

Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
designed it, I expect it would have used threads (without asyncore).
However, bits of code still sitting around suggest that it was at
least the _intent_ at one time that ZEO be able to run without threads
at all.  That's certainly not possible now.

If you look at asyncore's revision history, you'll note that Jeremy
and Guido made many changes when they worked at Zope Corp.  Those
largely reflect the history of moving ZEO's asyncore monkey-patches
into the Python core.

BTW, if you don't use ZEO, I believe it's possible to run Zope3
without asyncore (you can use Twisted in Zope3 instead).
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

"Martin v. Löwis"
Tim Peters wrote:
> Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
> designed it, I expect it would have used threads (without asyncore).
> However, bits of code still sitting around suggest that it was at
> least the _intent_ at one time that ZEO be able to run without threads
> at all.  That's certainly not possible now.

What is the reason that people want to use threads when they can have
poll/select-style message processing? Why does Zope require threads?
IOW, why would anybody *want* a "threadsafe patch for asynchat"?

Regards,
Martin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Steve Holden-5
Martin v. Löwis wrote:

> Tim Peters wrote:
>
>>Bugs and "missing features" in asyncore.  For ZEO's purposes, if I had
>>designed it, I expect it would have used threads (without asyncore).
>>However, bits of code still sitting around suggest that it was at
>>least the _intent_ at one time that ZEO be able to run without threads
>>at all.  That's certainly not possible now.
>
>
> What is the reason that people want to use threads when they can have
> poll/select-style message processing? Why does Zope require threads?
> IOW, why would anybody *want* a "threadsafe patch for asynchat"?
>
In case the processing of events needed to block? If I'm processing web
requests in an async* dispatch loop and a request needs the results of a
(probably lengthy) database query in order to generate its output, how
do I give the dispatcher control again to process the next asynchronous
network event?

The usual answer is "process the request in a thread". That way the
dispatcher can spring to life for each event as quickly as needed.

regards
  Steve
--
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC                     www.holdenweb.com
PyCon TX 2006                  www.python.org/pycon/

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Fredrik Lundh
Steve Holden wrote:

> > What is the reason that people want to use threads when they can have
> > poll/select-style message processing? Why does Zope require threads?
> > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> >
> In case the processing of events needed to block? If I'm processing web
> requests in an async* dispatch loop and a request needs the results of a
> (probably lengthy) database query in order to generate its output, how
> do I give the dispatcher control again to process the next asynchronous
> network event?
>
> The usual answer is "process the request in a thread". That way the
> dispatcher can spring to life for each event as quickly as needed.

but why do such threads have to talk to asyncore directly ?

</F>



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Josiah Carlson

"Fredrik Lundh" <[hidden email]> wrote:

>
> Steve Holden wrote:
>
> > > What is the reason that people want to use threads when they can have
> > > poll/select-style message processing? Why does Zope require threads?
> > > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> > >
> > In case the processing of events needed to block? If I'm processing web
> > requests in an async* dispatch loop and a request needs the results of a
> > (probably lengthy) database query in order to generate its output, how
> > do I give the dispatcher control again to process the next asynchronous
> > network event?
> >
> > The usual answer is "process the request in a thread". That way the
> > dispatcher can spring to life for each event as quickly as needed.
>
> but why do such threads have to talk to asyncore directly ?

Indeed.  I seem to remember a discussion a few months ago about "easy"
thread programming, which invariably directed people off to use the
simplest abstractions necessary: Queues.

 - Josiah

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Thomas Wouters-3
In reply to this post by jalopyuser
On Tue, Feb 07, 2006 at 08:53:46PM -0800, Bill Janssen wrote:

> Perhaps the right idea is to fix the various problems of asyncore.

The problem with making asyncore more useful is that you end up with (a cut
down version of) Twisted, although not one that would be able to integrate
with Twisted. asyncore/asynchat and Twisted are really not that different,
and anything you do to enhance the former will make it look more like the
latter. I'd personally rather fork parts of Twisted, in spite of the
maintenance issues, than re-invent Twisted, fix all the issues Twisted
already solves and face the same kind of maintenance issues. It would be
perfect if the twisted-light in the stdlib would integrate with the 'real'
Twisted, so that users can 'upgrade' their programs just by installing
Twisted and using the extra features.

Not that I think we should stop at the event core and the TCP/SSL parts of
Twisted; imaplib, poplib, httplib, xmlrpclib, for instance, could all do
with Twisted-inspired alternatives (or even replacements, if the synchronous
API was kept the same.) The synchronous versions are fine for simple scripts
(or complex scripts that don't mind long blocking operations.) If we start
exporting a really useful asynchronous framework, I would expect
asynchronous counterparts to the useful higher-level networking modules,
too. But that doesn't have to come right away ;)

Anything beyond simple bugfixes on asyncore/asynchat seems like a terrible
waste of effort, to me. And I hardly ever use Twisted.

--
Thomas Wouters <[hidden email]>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Steve Holden-5
In reply to this post by Josiah Carlson
Josiah Carlson wrote:

> "Fredrik Lundh" <[hidden email]> wrote:
>
>>Steve Holden wrote:
>>
>>
>>>>What is the reason that people want to use threads when they can have
>>>>poll/select-style message processing? Why does Zope require threads?
>>>>IOW, why would anybody *want* a "threadsafe patch for asynchat"?
>>>>
>>>
>>>In case the processing of events needed to block? If I'm processing web
>>>requests in an async* dispatch loop and a request needs the results of a
>>>(probably lengthy) database query in order to generate its output, how
>>>do I give the dispatcher control again to process the next asynchronous
>>>network event?
>>>
>>>The usual answer is "process the request in a thread". That way the
>>>dispatcher can spring to life for each event as quickly as needed.
>>
>>but why do such threads have to talk to asyncore directly ?
>
Good question.
>
> Indeed.  I seem to remember a discussion a few months ago about "easy"
> thread programming, which invariably directed people off to use the
> simplest abstractions necessary: Queues.
>
Maybe people are finding Python too easy and they just want to
complicate their code to the point where it contains interesting bugs? I
dunno ....

regards
  Steve
--
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC                     www.holdenweb.com
PyCon TX 2006                  www.python.org/pycon/

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: threadsafe patch for asynchat

Donovan Baarda
In reply to this post by Steve Holden-5
On Wed, 2006-02-08 at 02:33 -0500, Steve Holden wrote:
> Martin v. Löwis wrote:
> > Tim Peters wrote:
[...]

> > What is the reason that people want to use threads when they can have
> > poll/select-style message processing? Why does Zope require threads?
> > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> >
> In case the processing of events needed to block? If I'm processing web
> requests in an async* dispatch loop and a request needs the results of a
> (probably lengthy) database query in order to generate its output, how
> do I give the dispatcher control again to process the next asynchronous
> network event?
>
> The usual answer is "process the request in a thread". That way the
> dispatcher can spring to life for each event as quickly as needed.

I believe that Twisted does pretty much this with it's "deferred" stuff.
It shoves slow stuff off for processing in a separate thread that
re-syncs with the event loop when it's finished.

In the case of Zope/ZEO I'm not entirely sure but I think what happened
was medusa (asyncore/asynchat based stuff Zope2 was based on) didn't
have this deferred handler support. When they found some of the stuff
Zope was doing took a long time, they came up with an initially simpler
but IMHO uglier solution of running multiple async loops in separate
threads and using a front-end dispatcher to distribute connections to
them. This way it wasn't too bad if an async loop stalled, because the
other loops in other threads could continue to process stuff.

If ZEO is still using this approach I think switching to a twisted style
approach would be a good idea. However, I suspect this would be a very
painful refactor...

--
Donovan Baarda <[hidden email]>
http://minkirri.apana.org.au/~abo/

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12