[PEP 3148] futures - execute computations asynchronously

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

[PEP 3148] futures - execute computations asynchronously

Brian Quinlan
Hi all,

I recently submitted a daft PEP for a package designed to make it  
easier to execute Python functions asynchronously using threads and  
processes. It lets the user focus on their computational problem  
without having to build explicit thread/process pools and work queues.

The package has been discussed on stdlib-sig but now I'd like this  
group's feedback.

The PEP lives here:
http://python.org/dev/peps/pep-3148/

Here are two examples to whet your appetites:

"""Determine if several numbers are prime."""
import futures
import math

PRIMES = [
     112272535095293,
     112582705942171,
     112272535095293,
     115280095190773,
     115797848077099,
     1099726899285419]

def is_prime(n):
     if n % 2 == 0:
         return False

     sqrt_n = int(math.floor(math.sqrt(n)))
     for i in range(3, sqrt_n + 1, 2):
         if n % i == 0:
             return False
     return True

# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
     for number, is_prime in zip(PRIMES, executor.map(is_prime,  
PRIMES)):
         print('%d is prime: %s' % (number, is_prime))


"""Print out the size of the home pages of various new sites (and Fox  
News)."""
import futures
import urllib.request

URLS = ['http://www.foxnews.com/',
         'http://www.cnn.com/',
         'http://europe.wsj.com/',
         'http://www.bbc.co.uk/',
         'http://some-made-up-domain.com/']

def load_url(url, timeout):
     return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(max_workers=5) as executor:
     # Create a future for each URL load.
     future_to_url = dict((executor.submit(load_url, url, 60), url)
                          for url in URLS)

     # Iterate over the futures in the order that they complete.
     for future in futures.as_completed(future_to_url):
         url = future_to_url[future]
         if future.exception() is not None:
             print('%r generated an exception: %s' % (url,
                                                       
future.exception()))
         else:
             print('%r page is %d bytes' % (url, len(future.result())))

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Calvin Spealman
A young library solving an old problem in a way that conflicts with
many of the other implementations available for years and with zero
apparent users in the wild is not an appropriate candidate for a PEP.

On Fri, Mar 5, 2010 at 1:03 AM, Brian Quinlan <[hidden email]> wrote:

> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it easier to
> execute Python functions asynchronously using threads and processes. It lets
> the user focus on their computational problem without having to build
> explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this group's
> feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
>    112272535095293,
>    112582705942171,
>    112272535095293,
>    115280095190773,
>    115797848077099,
>    1099726899285419]
>
> def is_prime(n):
>    if n % 2 == 0:
>        return False
>
>    sqrt_n = int(math.floor(math.sqrt(n)))
>    for i in range(3, sqrt_n + 1, 2):
>        if n % i == 0:
>            return False
>    return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
>    for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
>        print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and Fox
> News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
>    # Create a future for each URL load.
>    future_to_url = dict((executor.submit(load_url, url, 60), url)
>                         for url in URLS)
>
>    # Iterate over the futures in the order that they complete.
>    for future in futures.as_completed(future_to_url):
>        url = future_to_url[future]
>        if future.exception() is not None:
>            print('%r generated an exception: %s' % (url,
>                                                     future.exception()))
>        else:
>            print('%r page is %d bytes' % (url, len(future.result())))
>
> Cheers,
> Brian
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>



--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman <[hidden email]> wrote:
> A young library solving an old problem in a way that conflicts with
> many of the other implementations available for years and with zero
> apparent users in the wild is not an appropriate candidate for a PEP.
>

Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package - but Brian's one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it's good
sugar, and is used by a fair number of people.

Let me also state - "my" vision of where futures would live would be
in a concurrent package - for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.

In any case; consider me a strong +1 to adding it.

jesse
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Calvin Spealman
I revert my objections. I still would like to see this in use "in the
wild" and I might even use it thusly, myself.

On Fri, Mar 5, 2010 at 9:50 AM, Jesse Noller <[hidden email]> wrote:

> On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman <[hidden email]> wrote:
>> A young library solving an old problem in a way that conflicts with
>> many of the other implementations available for years and with zero
>> apparent users in the wild is not an appropriate candidate for a PEP.
>>
>
> Baloney. A young library providing some syntactic sugar which uses
> primitives in the standard library to implement a common pattern is
> fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
> list prior to bringing it here. By the same argument, we should shunt
> all of the recent unittest changes and improvements into space, since
> golly, other people did it, why should we.
>
> This is something relatively simple, which I would gladly add in an
> instant to the multiprocessing package - but Brian's one-upped me in
> that regard and is providing something which works with both threads
> and processes handily. Take a look at multiprocessing.Pool for example
> - all that is some sugar on top of the primitives, but it's good
> sugar, and is used by a fair number of people.
>
> Let me also state - "my" vision of where futures would live would be
> in a concurrent package - for example:
>
> from concurrent import futures
>
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.
>
> For example, how many of us have sat down and implemented a thread
> pool on top of threading, I would hazard to say that most of us who
> use threading have done this, and probably more than once. It stands
> to reason that this is a common enough pattern to include in the
> standard library.
>
> In any case; consider me a strong +1 to adding it.
>
> jesse
>



--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Jesse Noller
Jesse Noller wrote:
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.

Yes, I've often thought we should have a pool model that covers threads
as well as processes.

The reason I think "futures" works as a PEP and potential standard
library addition is that it is small enough to be readily reviewed and
maintained, and serves as a useful building block for more complex usage.

For a developer to get anything similar from a third party library is
almost certainly going to require buying into a much heavier framework.
A simple futures module provides a way to farm out worker tasks in a
standard fashion without having to build as much of your own
infrastructure every time.

I've read the various PEP checkins as they went by on the checkins list
- it gets a +0 from me (the only reason it isn't a +1 is because I
personally tend to write with a Thread+Queue style. However, I could
easily become a futures convert if they were readily available in the
standard library)

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Daniel Stutzbach-2
In reply to this post by Brian Quinlan
On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
import futures

+1 on the idea, -1 on the name.  It's too similar to "from __future__ import ...".

Also, the PEP should probably link to the discussions on stdlib-sig?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
<[hidden email]> wrote:
> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
>>
>> import futures
>
> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
> ...".

Futures is a common term for this, and implemented named this in other
languages. I don't think we should be adopting things that are common,
and found elsewhere and then renaming them.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Skip Montanaro-3
    >>> import futures
    >>
    >> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
    >> ...".

    Jesse> Futures is a common term for this, and implemented named this in
    Jesse> other languages. I don't think we should be adopting things that
    Jesse> are common, and found elsewhere and then renaming them.

Perhaps, but is it a common term for Python programmers (or the target
population for Python)?  I've never heard of it.  "futures" to me are
futures contracts in a trading environment (that's the industry I work in).
No matter how well known the term is in the environment where it's used
today you have to be sensitive to other meanings of the term.

Skip

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
On Fri, Mar 5, 2010 at 11:56 AM,  <[hidden email]> wrote:

>    >>> import futures
>    >>
>    >> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>    >> ...".
>
>    Jesse> Futures is a common term for this, and implemented named this in
>    Jesse> other languages. I don't think we should be adopting things that
>    Jesse> are common, and found elsewhere and then renaming them.
>
> Perhaps, but is it a common term for Python programmers (or the target
> population for Python)?  I've never heard of it.  "futures" to me are
> futures contracts in a trading environment (that's the industry I work in).
> No matter how well known the term is in the environment where it's used
> today you have to be sensitive to other meanings of the term.
>

It's a common programming term. I don't think we should make a new
name - I mean, how many different names for green threads / coroutines
or "flavors" of those concepts are out there because someone painted
the bike shed a different color? I mean, there's prior art here:

http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

jesse
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Curt Hagenlocher
In reply to this post by Jesse Noller
On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller <[hidden email]> wrote:

>
> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
> <[hidden email]> wrote:
> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
> >>
> >> import futures
> >
> > +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
> > ...".
>
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

Another common term for this is a "promise".

--
Curt Hagenlocher
[hidden email]
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Daniel Stutzbach-2
In reply to this post by Jesse Noller
On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <[hidden email]> wrote:
http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

 According to that link, Java has a module named "Concurrent" with an interface named "Future".  You're proposing a module named "Futures" with a class named "Future".

Why not name your module "concurrent"?  That would eliminate the confusion with "from __future__".  I don't see a problem with keeping the class name.

Plus, a "concurrent" module might be useful for things other than Futures, in the future. ;-)

Just my 0.02 cents,
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Skip Montanaro-3
In reply to this post by Jesse Noller

    Jesse> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

Without reading that I can assure you that not everybody has drunk the Java
Kool-Aid.  Just because Sun thought it was a fine term doesn't mean everyone
else will.  I've been a professional programmer for about 30 years, have
never touched Java, and the only times I've seen its effect in Python I have
been unimpressed (logging and unittest).

Google for "future".  The Java doc is aways down on the front page.  Now
Google for "futures". It's nowhere to be found.  As far as I can tell, all
but one hit on the first page are for my definition of "futures".  At the
bottom of the page the "searches related to _futures_" are all related to
trading futures or other derivatives except one.  That appears to be related
to the LPGA futures tour.  Java and its definition of the term is nowhere to
be found.

I suppose you can argue that it should be called "future".  I still contend
that the name is *not* intuitive, no matter how well known it happens to be
within the Java world.  Don't name it "futures" though.  That has nothing to
do with common definitions of the term.

Skip
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brett Cannon-2
In reply to this post by Daniel Stutzbach-2


On Fri, Mar 5, 2010 at 09:55, Jesse Noller <[hidden email]> wrote:
On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
<[hidden email]> wrote:
> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <[hidden email]> wrote:
>>
>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>
>  According to that link, Java has a module named "Concurrent" with an
> interface named "Future".  You're proposing a module named "Futures" with a
> class named "Future".
>
> Why not name your module "concurrent"?  That would eliminate the confusion
> with "from __future__".  I don't see a problem with keeping the class name.
>
> Plus, a "concurrent" module might be useful for things other than Futures,
> in the future. ;-)
>

Brian's module is named futures; I am +1'ing his proposal, and also
suggesting we put it under concurrent/ package name. This means you
would do the following:

from concurrent import futures

and in the future:
from concurrent import pool

And so on.


So I don't quite get what you are after here. Are you wanting to eventually have a generic pool class that you can simply import and use that is always set to the best option for the platform?

And as for moving stuff from multiprocessing into the concurrent namespace, are you thinking like concurrent.multiprocessing? I guess I am just trying to figure out what the abstraction is you are after in the package namespace.

-Brett

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Tres Seaver
In reply to this post by Jesse Noller
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Noller wrote:

> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
> <[hidden email]> wrote:
>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
>>> import futures
>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>> ...".
>
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

- -1 to the name from me as well:  it isn't "scoped" properly to make it
clear what the module is about.  If they were inside a pacakge named
'concurrency' or some such (as hinted by Jesse Noller, I think), the
clash would go away.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          [hidden email]
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRapAACgkQ+gerLs4ltQ7dBwCfRmMuq6X9VE8usYgSScXEA1D0
1PsAoI8MR5hjPjq9C7MFPTZhcO/T+NM4
=7wpK
-----END PGP SIGNATURE-----

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brett Cannon-2
In reply to this post by Brian Quinlan
The PEP says that futures.wait() should only use keyword arguments past its first positional argument, but the PEP has the function signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.  Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?

On Thu, Mar 4, 2010 at 22:03, Brian Quinlan <[hidden email]> wrote:
Hi all,

I recently submitted a daft PEP for a package designed to make it easier to execute Python functions asynchronously using threads and processes. It lets the user focus on their computational problem without having to build explicit thread/process pools and work queues.

The package has been discussed on stdlib-sig but now I'd like this group's feedback.

The PEP lives here:
http://python.org/dev/peps/pep-3148/

Here are two examples to whet your appetites:

"""Determine if several numbers are prime."""
import futures
import math

PRIMES = [
   112272535095293,
   112582705942171,
   112272535095293,
   115280095190773,
   115797848077099,
   1099726899285419]

def is_prime(n):
   if n % 2 == 0:
       return False

   sqrt_n = int(math.floor(math.sqrt(n)))
   for i in range(3, sqrt_n + 1, 2):
       if n % i == 0:
           return False
   return True

# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
   for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
       print('%d is prime: %s' % (number, is_prime))


"""Print out the size of the home pages of various new sites (and Fox News)."""
import futures
import urllib.request

URLS = ['http://www.foxnews.com/',
       'http://www.cnn.com/',
       'http://europe.wsj.com/',
       'http://www.bbc.co.uk/',
       'http://some-made-up-domain.com/']

def load_url(url, timeout):
   return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(max_workers=5) as executor:
   # Create a future for each URL load.
   future_to_url = dict((executor.submit(load_url, url, 60), url)
                        for url in URLS)

   # Iterate over the futures in the order that they complete.
   for future in futures.as_completed(future_to_url):
       url = future_to_url[future]
       if future.exception() is not None:
           print('%r generated an exception: %s' % (url,
                                                    future.exception()))
       else:
           print('%r page is %d bytes' % (url, len(future.result())))

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
In reply to this post by Brett Cannon-2
On Fri, Mar 5, 2010 at 3:31 PM, Brett Cannon <[hidden email]> wrote:

>
> So I don't quite get what you are after here. Are you wanting to eventually
> have a generic pool class that you can simply import and use that is always
> set to the best option for the platform?
> And as for moving stuff from multiprocessing into the concurrent namespace,
> are you thinking like concurrent.multiprocessing? I guess I am just trying
> to figure out what the abstraction is you are after in the package
> namespace.
> -Brett

My goal would be to put futures into a "concurrent" package - as it is
an abstraction that allows for threads, and processes to be used. By
default; I don't think we'd make a guess based on the platform, but
rather pick a sane default (such as threads).

After that; I would begin to remove chunks of multiprocessing (such as
pool) and adapt it to the same type of "pick threads or processes"
that futures uses. For example:

from concurrent import Pool

 x = Pool(4, worker_primitive=Thread())

And so on. The end-goal would be to make concurrent.* a package
containing common abstractions/patterns which can use threads or
processes interchangeably.

jesse
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Guido van Rossum
In reply to this post by Curt Hagenlocher
On Fri, Mar 5, 2010 at 12:18 PM,  <[hidden email]> wrote:
> The "explicit" futures on the wikipedia page seems to cover what is commonly
> referred to as a future.  For example, Java's futures look like this.
>
> The "implicit" futures are what is generally called a promise.  For example,
> E's promises look like this.

Fair enough, though the article confuses the matter by using the words
more or less interchangeably.

> Though the difference is mainly one of API, it turns out to make a
> significant difference in what you can accomplish.  Promises are much more
> amenable to the pipelining optimization, for example.  They're also much
> harder to implement in Python without core language changes.

That's why implicit futures (by any name) aren't on the table.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
In reply to this post by Tres Seaver
On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jesse Noller wrote:
>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>> <[hidden email]> wrote:
>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
>>>> import futures
>>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>>> ...".
>>
>> Futures is a common term for this, and implemented named this in other
>> languages. I don't think we should be adopting things that are common,
>> and found elsewhere and then renaming them.
>
> - -1 to the name from me as well:  it isn't "scoped" properly to make it
> clear what the module is about.  If they were inside a pacakge named
> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
> clash would go away.

If people agree with this; do you feel the proposal of said namespace
should be a separate PEP, or piggy back on this? I don't want to piggy
back on Brian's hard work.

Jesse
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Guido van Rossum
On Fri, Mar 5, 2010 at 1:42 PM, Jesse Noller <[hidden email]> wrote:

> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <[hidden email]> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>> <[hidden email]> wrote:
>>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]> wrote:
>>>>> import futures
>>>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>>>> ...".
>>>
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>>
>> - -1 to the name from me as well:  it isn't "scoped" properly to make it
>> clear what the module is about.  If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
>
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

A simple renaming of futures to concurrency.futures seems easy enough
to swallow. (Though I haven't kept track of what other modules the PEP
proposes.)

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Antoine Pitrou
In reply to this post by Brian Quinlan
Le Fri, 5 Mar 2010 17:03:02 +1100,
Brian Quinlan <[hidden email]> a écrit :
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/

Ok, here is my take on it:

> cancel()
>
> Attempt to cancel the call. If the call is currently being executed
> then it cannot be cancelled and the method will return False,
> otherwise the call will be cancelled and the method will return
> True.

I think it shouldn't return anything, and raise an exception if
cancelling failed. It is really an error condition, and ignoring the
result doesn't seem right.

> Future.running()
>
> Return True if the call is currently being executed and cannot be
> cancelled.
>
> Future.done()
>
> Return True if the call was successfully cancelled or finished
> running.

These don't really make sense since the future is executing
concurrently. By the time the result is returned, it can already be
wrong. I advocate removing those two methods.

> The following Future methods are meant for use in unit tests and
> Executor implementations.

Their names should then be preceded by an underscore '_'. We don't want
people to think they are public APIs and start relying on them.

> wait(fs, timeout=None, return_when=ALL_COMPLETED)
> [...]
>
> This method should always be called using keyword arguments

I don't think this is right. Keyword arguments are nice, but mandating
them too often is IMO a nuisance (after all, it makes things longer to
type and requires you to remember the exact parameter names).
Especially when the method only takes at most 3 arguments.

IMO, keyword-only arguments are mostly useful when there are a lot of
positional arguments before, and you want to help the user use the
right calling signature.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12345