[PEP 3148] futures - execute computations asynchronously

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Tres Seaver
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Noller wrote:

> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <[hidden email]> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>> <[hidden email]> wrote:
>>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
>>>>> import futures
>>>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>>>> ...".
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>> - -1 to the name from me as well:  it isn't "scoped" properly to make it
>> clear what the module is about.  If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
>
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

I'm just expressiong a preference for scoping the name, and don't want
to preempt the process.  If your proposed work on factoring common stuff
out of multiprocessing would sit in the same conceptual space, then
sharing the package name seems like a good plan to me.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          [hidden email]
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRj74ACgkQ+gerLs4ltQ5RNACeKYku88A9PBuQR46QTl7GrEwo
mPEAoLdYyi+TLGYFw4SRAIM8zBsNvwxr
=iPkb
-----END PGP SIGNATURE-----

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jeffrey Yasskin-2
In reply to this post by Antoine Pitrou
On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou <[hidden email]> wrote:

> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan <[hidden email]> a écrit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

The caller can't avoid the error here by querying the future, because
of the problem you point out below, so I'm inclined to think that "the
future was already started" should be a return value rather than an
exception (although that may be my C++ background showing through).
Would calling the method try_cancel() work better?

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

"done()" can only be wrong in one direction though. If it returns
True, it'll never again return False, which can be useful (although
perhaps only for polling, which might be an argument for removing it
anyway).

"running()" becomes True and then False again, so I agree with your
objection. A "started()" function would only go from False to True
once. Maybe that's a better function?

Jeffrey
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Stephen J. Turnbull
In reply to this post by Daniel Stutzbach-2
Guido van Rossum writes:

 > "Future" is a pretty standard CS term for this concept (as noted
 > "promise" is another),

I like the term "promise" better.  "Future" is very generic ("not now,
but later"), whereas a "promise" is something I don't get from you
now, but you will give me later.

The wikipedia article is not very helpful on the implicit vs. explicit
distinction.  As far as I can tell from it, that distinction isn't
really attached to "future" vs "promise."  The only distinction the
article described was in the context of the Alice language, where a
future = promise (read-only) plus resolver (mutator).  IMO that's not
a compelling reason for adopting "future" in Python.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Stephen J. Turnbull
In reply to this post by Curt Hagenlocher
[hidden email] writes:

 > The "explicit" futures on the wikipedia page seems to cover what is
 > commonly referred to as a future.  For example, Java's futures look like
 > this.
 >
 > The "implicit" futures are what is generally called a promise.  For
 > example, E's promises look like this.

*sigh*  All I can say is "it's a damned shame that there are no native
speakers of English working in computer science."<wink>

I have to admit Jean-Paul's explanation a pretty convincing reason for
adopting "future" rather than "promise".  But I'm with Skip, I would
prefer that the module be named "future" rather than "futures".
(Especially when wearing my Professional Economist sweatshirt. :-)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
In reply to this post by Brian Quinlan
At 01:03 AM 3/5/2010, Brian Quinlan wrote:
>Hi all,
>
>I recently submitted a daft PEP for a package designed to make it
>easier to execute Python functions asynchronously using threads and
>processes. It lets the user focus on their computational problem
>without having to build explicit thread/process pools and work queues.
>
>The package has been discussed on stdlib-sig but now I'd like this
>group's feedback.

My immediate reaction is that this would be a lot more useful if it
built on an API for coroutine yielding/interaction, similar to what's
in say, Eventlet.  That would seem to make it easier to write
synchronous-looking code that operates on futures, and allow futures
to be composed more cleanly.

ISTM that if futures were standardized before a coroutine API, it
would lead to effective orphaning ala what happened with asyncore,
especially since the new library is, well, new.

I'm somewhat concerned that, as described, the proposed API adds
little over what's relatively easy to do with a mature coroutine
framework like Eventlet, while at the same time creating yet another
alternative (and mutually incompatible) event loop system in the
stdlib, beyond the ones that are already in asyncore, tkinter, and
the various SocketServer subclasses.

As far as naming goes, Twisted uses the term "Deferred" for this
concept (and also has a very mature API for handling them).

And speaking of Twisted, it seems to me that the PEP would be much
improved in general by learning from some of the lessons of other
systems.  I don't think that Java's example is really the best one to
follow in this instance, compared to the many existing async
frameworks that have Python-specific experience and APIs to learn from.

Or, to put it another way, something that worries me about this PEP
is that nearly all of its Python-related citations are for
*discussions* of futures, with the only previous Python
implementation cited being a crude sketch of a cookbook recipe.  The
PEP also doesn't address questions of interoperability with existing
solutions, compare features with them, or even so much as say, "There
are other production implementations of this concept in Python, but
we are going to pretend they don't exist."  ;-)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jeffrey Yasskin-2
On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <[hidden email]> wrote:
> I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and
> mutually incompatible) event loop system in the stdlib ...

Futures are a blocking construct; they don't involve an event loop.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:
>On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <[hidden email]> wrote:
> > I'm somewhat concerned that, as described, the proposed API ...
> [creates] yet another alternative (and
> > mutually incompatible) event loop system in the stdlib ...
>
>Futures are a blocking construct; they don't involve an event loop.

And where they block is in a loop, waiting for events (completed
promises) coming back from other threads or processes.

The Motivation section of the PEP also stresses avoiding reinvention
of such loops, and points to the complication of using more than one
at a time as a justification for the mechanism.  It seems relevant to
at least address why wrapping multiprocessing and multithreading is
appropriate, but *not* dealing with any other form of sync/async
boundary, *or* composition of futures.

On which subject, I might add, the PEP is silent on whether executors
are reentrant to the called code.  That is, can I call a piece of
code that uses futures, using the futures API?  How will the called
code know what executor to use?  Must I pass it one explicitly?  Will
that work across threads and processes, without explicit support from the API?

IOW, as far as I can tell from the PEP, it doesn't look like you can
compose futures without *global* knowledge of the application...  and
in and of itself, this seems to negate the PEP's own motivation to
prevent duplication of parallel execution handling!

That is, if I use code from module A and module B that both want to
invoke tasks asynchronously, and I want to invoke A and B
asynchronously, what happens?  Based on the design of the API, it
appears there is nothing you can do except refactor A and B to take
an executor in a parameter, instead of creating their own.

It seems therefore to me that either the proposal does not define its
scope/motivation very well, or it is not well-equipped to address the
problem it's setting out to solve.  If it's meant to be something
less ambitious -- more like a recipe or example -- it should properly
motivate that scope.  If it's intended to be a robust tool for
composing different pieces of code, OTOH, it should absolutely
address the issue of writing composable code...  since, that seems to
be what it says the purpose of the API is.  (I.e., composing code to
use a common waiting loop.)

And, existing Python async APIs (such as Twisted's Deferreds)
actually *address* this issue of composition; the PEP does
not.  Hence my comments about not looking at existing implementations
for API and implementation guidance.  (With respect to what the API
needs, and how it needs to do it, not necessarily directly copying
actual APIs or implementations.  Certainly some of the Deferred API
naming has a rather, um, "twisted" vocabulary.)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jeffrey Yasskin-2
In reply to this post by Stephen J. Turnbull
On Fri, Mar 5, 2010 at 9:47 PM, Stephen J. Turnbull <[hidden email]> wrote:

> Guido van Rossum writes:
>
>  > "Future" is a pretty standard CS term for this concept (as noted
>  > "promise" is another),
>
> I like the term "promise" better.  "Future" is very generic ("not now,
> but later"), whereas a "promise" is something I don't get from you
> now, but you will give me later.
>
> The wikipedia article is not very helpful on the implicit vs. explicit
> distinction.  As far as I can tell from it, that distinction isn't
> really attached to "future" vs "promise."  The only distinction the
> article described was in the context of the Alice language, where a
> future = promise (read-only) plus resolver (mutator).  IMO that's not
> a compelling reason for adopting "future" in Python.

It seems like a good idea to follow the choice other languages have
used for the name (if they tend to agree) regardless of whether the
evil Java followed it too. So let's take a poll:

Io: Uses "future" to refer to the implicit kind
(http://www.iolanguage.com/scm/io/docs/IoGuide.html#Concurrency-Futures)
Alice ML: Uses "future" to refer to the implicit kind, and "promise"
to refer to a handle that can fill in the future
(http://www.ps.uni-saarland.de/alice/manual/futures.html)
Java: Uses "future" to refer to the explicit kind.
(http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html)
AmbientTalk: Uses "future" to refer to something more like a deferred:
you register callbacks to run when the future is resolved.
(http://soft.vub.ac.be/amop/at/tutorial/actors#futures)
C++0x: Uses "future" to refer to the explicit kind; "promise"
similarly to AliceML, and "packaged_task" to get a future from a
callable. (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3035.pdf
section 30.6)
E: Uses "promise" to refer to the implicit kind, and "resolver" to
refer to a handle that can fill in the promise.
(http://wiki.erights.org/wiki/Promise)
Oz: A "future" is a read-only logic variable. I'm not entirely sure
what that means.
(http://www.mozart-oz.org/documentation/dstutorial/node2.html#label61)
PLT Scheme: Uses "future" to refer to the explicit kind.
(http://docs.plt-scheme.org/futures/index.html)
C#: Uses "Task<TResult>" to refer to the explicit kind.
(http://msdn.microsoft.com/en-us/library/dd321424(VS.100).aspx)
Id, from 1991: Used "I-structure" to refer to the implicit kind.
(http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4920)
Scala: Uses "Future" to refer to the explicit kind.
(http://www.scala-lang.org/docu/files/api/scala/actors/Actor.html#%21%21%28Any%2CPartialFunction%5BAny%2CA%5D%29)

What languages did I miss? From this list, "future" seems to be the
most popular choice, and it doesn't seem to distinguish between the
implicit and explicit kinds.

Jeffrey
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Daniel Stutzbach-2

On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:

On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
import futures

+1 on the idea, -1 on the name.  It's too similar to "from __future__ import ...".

Also, the PEP should probably link to the discussions on stdlib-sig?

I thought about that but this discussion is spread over many threads and many months.

Cheers,
Brian


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Brett Cannon-2
On 6 Mar 2010, at 07:38, Brett Cannon wrote:

The PEP says that futures.wait() should only use keyword arguments past its first positional argument, but the PEP has the function signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.  Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?

Hi Brett,

That recommendation was designed to make it easy to change the API without breaking code.

I'd don't think that recommendation makes sense anymore any I'll update the PEP.

Cheers,
Brian

On Thu, Mar 4, 2010 at 22:03, Brian Quinlan <[hidden email]> wrote:
Hi all,

I recently submitted a daft PEP for a package designed to make it easier to execute Python functions asynchronously using threads and processes. It lets the user focus on their computational problem without having to build explicit thread/process pools and work queues.

The package has been discussed on stdlib-sig but now I'd like this group's feedback.

The PEP lives here:
http://python.org/dev/peps/pep-3148/

Here are two examples to whet your appetites:

"""Determine if several numbers are prime."""
import futures
import math

PRIMES = [
   112272535095293,
   112582705942171,
   112272535095293,
   115280095190773,
   115797848077099,
   1099726899285419]

def is_prime(n):
   if n % 2 == 0:
       return False

   sqrt_n = int(math.floor(math.sqrt(n)))
   for i in range(3, sqrt_n + 1, 2):
       if n % i == 0:
           return False
   return True

# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
   for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
       print('%d is prime: %s' % (number, is_prime))


"""Print out the size of the home pages of various new sites (and Fox News)."""
import futures
import urllib.request

URLS = ['http://www.foxnews.com/',
       'http://www.cnn.com/',
       'http://europe.wsj.com/',
       'http://www.bbc.co.uk/',
       'http://some-made-up-domain.com/']

def load_url(url, timeout):
   return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(max_workers=5) as executor:
   # Create a future for each URL load.
   future_to_url = dict((executor.submit(load_url, url, 60), url)
                        for url in URLS)

   # Iterate over the futures in the order that they complete.
   for future in futures.as_completed(future_to_url):
       url = future_to_url[future]
       if future.exception() is not None:
           print('%r generated an exception: %s' % (url,
                                                    future.exception()))
       else:
           print('%r page is %d bytes' % (url, len(future.result())))

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Jesse Noller

On 6 Mar 2010, at 08:42, Jesse Noller wrote:
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

It doesn't really matter to me.

We can either update this PEP to propose the concurrent.futures name  
or you can draft a more complete PEP that describes what other  
functionality should live in the concurrent package.

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Antoine Pitrou

On 6 Mar 2010, at 09:54, Antoine Pitrou wrote:

> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan <[hidden email]> a écrit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

In my experience with futures, canceling them is a best-effort  
optimization that people use when another future fails. For example:

futures = [executor.submit(CopyDirectory, src, dest) for dest in ...]
finished, unfinished = wait(futures, return_when=FIRST_EXCEPTION)
# If there are unfinished futures then there must have been a failure
for f in unfinished:
   # No reason to waste bandwidth copying files if the operation has  
already failed.
   f.cancel()
for f in finished():
   if f.exception():
     raise f.exception()

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

There methods are useful for logging - by displaying the count of  
pending, running and completed futures you can estimate the progress  
of the system.

>> The following Future methods are meant for use in unit tests and
>> Executor implementations.
>
> Their names should then be preceded by an underscore '_'. We don't  
> want
> people to think they are public APIs and start relying on them.

Actually, as discussed on the stdlib-sig, these methods are designed  
to make it possible for users to implement their own Executors so  
we'll have keep the interface stable.

>> wait(fs, timeout=None, return_when=ALL_COMPLETED)
>> [...]
>>
>> This method should always be called using keyword arguments
>
> I don't think this is right. Keyword arguments are nice, but mandating
> them too often is IMO a nuisance (after all, it makes things longer to
> type and requires you to remember the exact parameter names).
> Especially when the method only takes at most 3 arguments.
>
> IMO, keyword-only arguments are mostly useful when there are a lot of
> positional arguments before, and you want to help the user use the
> right calling signature.

I agree, I'll change this.

Cheers,
Brian

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by PJ Eby

On 6 Mar 2010, at 17:50, Phillip J. Eby wrote:

> At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:
>> On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <[hidden email]
>> > wrote:
>> > I'm somewhat concerned that, as described, the proposed API ...  
>> [creates] yet another alternative (and
>> > mutually incompatible) event loop system in the stdlib ...
>>
>> Futures are a blocking construct; they don't involve an event loop.
>
> And where they block is in a loop, waiting for events (completed  
> promises) coming back from other threads or processes.
>
> The Motivation section of the PEP also stresses avoiding reinvention  
> of such loops, and points to the complication of using more than one  
> at a time as a justification for the mechanism.  It seems relevant  
> to at least address why wrapping multiprocessing and multithreading  
> is appropriate, but *not* dealing with any other form of sync/async  
> boundary, *or* composition of futures.
>
> On which subject, I might add, the PEP is silent on whether  
> executors are reentrant to the called code.  That is, can I call a  
> piece of code that uses futures, using the futures API?  How will  
> the called code know what executor to use?  Must I pass it one  
> explicitly?  Will that work across threads and processes, without  
> explicit support from the API?

Executors are reentrant but deadlock is possible. There are two  
deadlock examples in the PEP.

>
> IOW, as far as I can tell from the PEP, it doesn't look like you can  
> compose futures without *global* knowledge of the application...  
> and in and of itself, this seems to negate the PEP's own motivation  
> to prevent duplication of parallel execution handling!
>
> That is, if I use code from module A and module B that both want to  
> invoke tasks asynchronously, and I want to invoke A and B  
> asynchronously, what happens?  Based on the design of the API, it  
> appears there is nothing you can do except refactor A and B to take  
> an executor in a parameter, instead of creating their own.

A and B could both use their own executor instances. You would need to  
refactor A and B if you wanted to manage thread and process counts  
globally.

> It seems therefore to me that either the proposal does not define  
> its scope/motivation very well, or it is not well-equipped to  
> address the problem it's setting out to solve.  If it's meant to be  
> something less ambitious -- more like a recipe or example -- it  
> should properly motivate that scope.  If it's intended to be a  
> robust tool for composing different pieces of code, OTOH, it should  
> absolutely address the issue of writing composable code...  since,  
> that seems to be what it says the purpose of the API is.  (I.e.,  
> composing code to use a common waiting loop.)

My original motivation when designing this module was having to deal  
with a lot of code that looks like this:

def get_some_user_info(user):
   x = make_ldap_call1(user)
   y = make_ldap_call2(user)
   z = [make_db_call(user, i) for i in something]

   # Do some processing with x, y, z and return a result

Doing these operations serially is too slow. So how do I parallelize  
them? Using the threading module is the obvious choice but having to  
create my own work/result queue every time I encounter this pattern is  
annoying. The futures module lets you write this as:

def get_some_user_info(user):
   with ThreadPoolExecutor(max_threads=10) as executor:
     x_future = executor.submit(make_ldap_call1, user)
     y_future = executor.submit(make_ldap_call2, user)
     z_futures = [executor.submit(make_db_call, user, i) for i in  
something]
    finished, _ = wait([x_future, y_future] + z_futures,  
return_when=FIRST_EXCEPTION)
  for f in finished:
    if f.exception():
      raise f.exception()
  x = x_future.result()
  y = y_future.result()
  z = [f.result() for f in z_futures]

   # Do some processing with x, y, z and return a result

> And, existing Python async APIs (such as Twisted's Deferreds)  
> actually *address* this issue of composition; the PEP does not.  
> Hence my comments about not looking at existing implementations for  
> API and implementation guidance.  (With respect to what the API  
> needs, and how it needs to do it, not necessarily directly copying  
> actual APIs or implementations.  Certainly some of the Deferred API  
> naming has a rather, um, "twisted" vocabulary.)

Using twisted (or any other asynchronous I/O framework) forces you to  
rewrite your I/O code. Futures do not.

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Brian Quinlan
Brian Quinlan wrote:

>
> On 6 Mar 2010, at 08:42, Jesse Noller wrote:
>> If people agree with this; do you feel the proposal of said namespace
>> should be a separate PEP, or piggy back on this? I don't want to piggy
>> back on Brian's hard work.
>
> It doesn't really matter to me.
>
> We can either update this PEP to propose the concurrent.futures name or
> you can draft a more complete PEP that describes what other
> functionality should live in the concurrent package.

I think a "concurrent.futures" name works - it gives the scoping desired
by the folks with an finance background and gives us a bucket for future
thread/process agnostic concurrency tools (such as a pools and message
queues).

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Stephen J. Turnbull
In reply to this post by Jeffrey Yasskin-2
Jeffrey Yasskin writes:

 > It seems like a good idea to follow the choice other languages have
 > used for the name (if they tend to agree)

For the record, I've conceded that point.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Brian Quinlan
Brian Quinlan wrote:

>> IOW, as far as I can tell from the PEP, it doesn't look like you can
>> compose futures without *global* knowledge of the application...  and
>> in and of itself, this seems to negate the PEP's own motivation to
>> prevent duplication of parallel execution handling!
>>
>> That is, if I use code from module A and module B that both want to
>> invoke tasks asynchronously, and I want to invoke A and B
>> asynchronously, what happens?  Based on the design of the API, it
>> appears there is nothing you can do except refactor A and B to take an
>> executor in a parameter, instead of creating their own.
>
> A and B could both use their own executor instances. You would need to
> refactor A and B if you wanted to manage thread and process counts
> globally.

You may want to consider providing global thread and process executors
in the futures module itself. Code which just wants to say "do this in
the background" without having to manage the lifecycle of its own
executor instance is then free to do so. I've had a lot of experience
with a framework that provides this and it is *very* convenient (it's
also a good way to avoid deadlocks due to synchronous notification APIs).


On PJE's broader point, async event loops with non-blocking I/O and
messages passed back to the event loop to indicate completion of
operations and relying on threads and processes to farm out tasks (which
you will later block on in order to retrieve the results) are completely
different programming models. This PEP doesn't change that - it just
makes certain aspects of the latter approach easier to handle.

Trying to design an API that can cope with either model strikes me as a
fool's errand. They differ at such a fundamental level that I don't see
how a hybrid API could be particularly optimal for either approach.

Cheers,
Nick.


--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Daniel Stutzbach-2
On Sat, Mar 6, 2010 at 6:43 AM, Nick Coghlan <[hidden email]> wrote:
You may want to consider providing global thread and process executors
in the futures module itself. Code which just wants to say "do this in
the background" without having to manage the lifecycle of its own
executor instance is then free to do so.

+1
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
In reply to this post by Brian Quinlan
At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>Using twisted (or any other asynchronous I/O framework) forces you to
>rewrite your I/O code. Futures do not.

Twisted's "Deferred" API has nothing to do with I/O.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
I have been playing with the feedback branch of this package for py3
and there seems to be a rather serious bug in the Process version.
Using the code @ http://dpaste.com/hold/168795/

When I was running in debug mode I found that as soon as

            p = multiprocessing.Process(
                    target=_process_worker,
                    args=(self._call_queue,
                          self._result_queue,
                          self._shutdown_process_event))

was called (yes even before p.start() was called) the processes just
started launching all by themselves.

I am also wondering why you are launching the process directly instead
of using a Pool since you are limiting the number of processes always
wouldnt it be better to launch the worker processes up front then just
add worker items to the queue?
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
I am also getting an odd error on odd error on exit

Error in atexit._run_exitfuncs:
TypeError: print_exception(): Exception expected for value, str found
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12345