[PEP 3148] futures - execute computations asynchronously

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller


On Mar 6, 2010, at 4:20 PM, Dj Gilcrease <[hidden email]> wrote:

> I have been playing with the feedback branch of this package for py3
> and there seems to be a rather serious bug in the Process version.
> Using the code @ http://dpaste.com/hold/168795/
>
> When I was running in debug mode I found that as soon as
>
>            p = multiprocessing.Process(
>                    target=_process_worker,
>                    args=(self._call_queue,
>                          self._result_queue,
>                          self._shutdown_process_event))
>
> was called (yes even before p.start() was called) the processes just
> started launching all by themselves.
>

Did you run the provided example code on windows by chance? If so,  
look at the multiprocessing docs, there are restrictions on windows  
(see the __main__ note) - not following the guidelines can result in  
lots of processes spawning.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Ben Finney-10
In reply to this post by Stephen J. Turnbull
"Stephen J. Turnbull" <[hidden email]> writes:

> I have to admit Jean-Paul's explanation a pretty convincing reason for
> adopting "future" rather than "promise". But I'm with Skip, I would
> prefer that the module be named "future" rather than "futures".

Has anyone in this very long thread raised the issue that Python
*already* uses this term for the name of a module with a totally
unrelated purpose; the ‘__future__’ pseudo-module?

That alone seems a pretty strong reason to avoid the word “future”
(singular or plural) for some other module name.

--
 \     “Creativity can be a social contribution, but only in so far as |
  `\            society is free to use the results.” —Richard Stallman |
_o__)                                                                  |
Ben Finney

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller


On Mar 6, 2010, at 5:47 PM, Ben Finney <[hidden email]>  
wrote:

> "Stephen J. Turnbull" <[hidden email]> writes:
>
>> I have to admit Jean-Paul's explanation a pretty convincing reason  
>> for
>> adopting "future" rather than "promise". But I'm with Skip, I would
>> prefer that the module be named "future" rather than "futures".
>
> Has anyone in this very long thread raised the issue that Python
> *already* uses this term for the name of a module with a totally
> unrelated purpose; the ‘__future__’ pseudo-module?
>
> That alone seems a pretty strong reason to avoid the word “future”
> (singular or plural) for some other module name.
>

Yes, they have, and putting it in a sub namespace has also come up. In  
the thread.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by Jeffrey Yasskin-2
Jeffrey Yasskin wrote:

> The caller can't avoid the error here by querying the future, because
> of the problem you point out below, so I'm inclined to think that "the
> future was already started" should be a return value rather than an
> exception (although that may be my C++ background showing through).

I think it's your C++ background showing. In Python philosophy,
there's no particlular connection between whether something can
be tested for and whether it should raise an exception.

The thing to consider, I think, is whether it makes sense in
a large proportion of use cases to ignore the fact that the
cancel failed and carry on regardless. If not, then raising an
exception makes it much harder to accidentally ignore the
situation.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by PJ Eby
Phillip J. Eby wrote:
> while at the same time creating yet another alternative (and
> mutually incompatible) event loop system in the stdlib, beyond the ones
> that are already in asyncore, tkinter, and the various SocketServer
> subclasses.

Aaargh... that's the *last* thing we need!

I've been thinking for a while that it would be a big help
if there were one, standardised module in the stdlib for
handling async events, and all the other gui toolkits
etc. were made to use it.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Michael Foord-5
On 06/03/2010 23:37, Greg Ewing wrote:

> Phillip J. Eby wrote:
>> while at the same time creating yet another alternative (and mutually
>> incompatible) event loop system in the stdlib, beyond the ones that
>> are already in asyncore, tkinter, and the various SocketServer
>> subclasses.
>
> Aaargh... that's the *last* thing we need!
>
> I've been thinking for a while that it would be a big help
> if there were one, standardised module in the stdlib for
> handling async events, and all the other gui toolkits
> etc. were made to use it.
>
Wouldn't it have to be the Tcl event loop then?

Michael

--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by Brian Quinlan
Brian Quinlan wrote:

> That recommendation was designed to make it easy to change the API
> without breaking code.
>
> I'd don't think that recommendation makes sense anymore any I'll update
> the PEP.

I don't think there's anything wrong with stating that the
order of the arguments is not a guaranteed part of the API.
There isn't necessarily any need to enforce that, though.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Daniel Stutzbach-2
In reply to this post by Michael Foord-5
On Sat, Mar 6, 2010 at 5:38 PM, Michael Foord <[hidden email]> wrote:
On 06/03/2010 23:37, Greg Ewing wrote:
I've been thinking for a while that it would be a big help
if there were one, standardised module in the stdlib for
handling async events, and all the other gui toolkits
etc. were made to use it.

Wouldn't it have to be the Tcl event loop then?

I image he means a standardized Abstract Base Class, which each GUI toolkit would subclass.  That's more or less how Twisted's "reactors" work.  That way non-GUI async code can just use the ABC and not worry about what event loop is running underneath (be it TCL, GTK, or just poll()).
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by Michael Foord-5
Michael Foord wrote:

> Wouldn't it have to be the Tcl event loop then?

No, tcl/tk would have to be threatened with the comfy chair
until it allowed itself to be spliced into the official
event loop somehow.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Skip Montanaro-3
In reply to this post by Ben Finney-10

    Ben> Has anyone in this very long thread raised the issue that Python
    Ben> *already* uses this term for the name of a module with a totally
    Ben> unrelated purpose; the ‘__future__’ pseudo-module?

Yes, it's already come up.  While not quite the same it does remind me of
the __builtin__/__builtins__ confusion.

Skip
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by Jesse Noller
On Sat, Mar 6, 2010 at 2:58 PM, Jesse Noller <[hidden email]> wrote:
> Did you run the provided example code on windows by chance? If so, look at
> the multiprocessing docs, there are restrictions on windows (see the
> __main__ note) - not following the guidelines can result in lots of
> processes spawning.


Yes, on win7. I might recommend then that the examples in the PEP be
restructured to work correctly on windows by including the

if __name__ == '__main__':
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by PJ Eby
On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:

> At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>> Using twisted (or any other asynchronous I/O framework) forces you to
>> rewrite your I/O code. Futures do not.
>
> Twisted's "Deferred" API has nothing to do with I/O.

I see, you just mean the API and not the underlying model.

We discussed the Deferred API on the stdlib-sig and I don't think that  
anyone expressed a preference for it over the one described in the PEP.

Do you have any concrete criticism?

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

exarkun
On 02:10 am, [hidden email] wrote:

>On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>>At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>Using twisted (or any other asynchronous I/O framework) forces you to
>>>rewrite your I/O code. Futures do not.
>>
>>Twisted's "Deferred" API has nothing to do with I/O.
>
>I see, you just mean the API and not the underlying model.
>
>We discussed the Deferred API on the stdlib-sig and I don't think that
>anyone expressed a preference for it over the one described in the PEP.
>
>Do you have any concrete criticism?

 From reading some of the stdlib-sig archives, it sounds like there is
general agreement that Deferreds and Futures can be used to complement
each other, and that getting code that is primarily Deferred-based to
integrate with Future-based code or vice versa should eventually be
possible.

Do I have the right sense of people's feelings?

And relatedly, once Futures are accepted and implemented, are people
going to use them as an argument to exclude Deferreds from the stdlib
(or be swayed by other people making such arguments)?  Hopefully not,
given what I read on stdlib-sig, but it doesn't hurt to check...

Jean-Paul
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
On Sat, Mar 6, 2010 at 9:34 PM,  <[hidden email]> wrote:

> On 02:10 am, [hidden email] wrote:
>>
>> On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>>>
>>> At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>>
>>>> Using twisted (or any other asynchronous I/O framework) forces you to
>>>> rewrite your I/O code. Futures do not.
>>>
>>> Twisted's "Deferred" API has nothing to do with I/O.
>>
>> I see, you just mean the API and not the underlying model.
>>
>> We discussed the Deferred API on the stdlib-sig and I don't think that
>> anyone expressed a preference for it over the one described in the PEP.
>>
>> Do you have any concrete criticism?
>
> From reading some of the stdlib-sig archives, it sounds like there is
> general agreement that Deferreds and Futures can be used to complement each
> other, and that getting code that is primarily Deferred-based to integrate
> with Future-based code or vice versa should eventually be possible.
>
> Do I have the right sense of people's feelings?
>
> And relatedly, once Futures are accepted and implemented, are people going
> to use them as an argument to exclude Deferreds from the stdlib (or be
> swayed by other people making such arguments)?  Hopefully not, given what I
> read on stdlib-sig, but it doesn't hurt to check...
>
> Jean-Paul

Generally speaking; I don't see futures as an exclusion to Deferreds,
or other asynchronous doodads. I just see it as a useful construct on
top of threads and processes primarily. So in my mind, no.

jesse
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by Brian Quinlan
After playing with the API for a while & running into many issues with
the examples & tests crashing windows I decided to modify the API a
little and fix up the examples so they dont crash windows based
computers.

http://code.google.com/p/pythonfutures/issues/detail?id=1

API Change that changes the current Executor to ExecutorBase and
adds a new Executor class that is used like

futures.Executor() # creates an executor that uses threading and a
max_workers = to the number of cpus

futures.Executor(use='process') # Creates an executor that uses
multiprocessing and a max_workers = to the number of cpus

futures.Executor(max_workers=5) # threading again, just specifying the
number of workers

futures.Executor(use='process', max_workers=5) # back to multiprocessing,
but with the max_workers specified
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
In reply to this post by Brian Quinlan
At 01:10 PM 3/7/2010 +1100, Brian Quinlan wrote:

>On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>
>>At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>Using twisted (or any other asynchronous I/O framework) forces you to
>>>rewrite your I/O code. Futures do not.
>>
>>Twisted's "Deferred" API has nothing to do with I/O.
>
>I see, you just mean the API and not the underlying model.
>
>We discussed the Deferred API on the stdlib-sig and I don't think that
>anyone expressed a preference for it over the one described in the PEP.
>
>Do you have any concrete criticism?

Of the PEP, yes, absolutely, and I've already stated much of it.  My
quibbles are with the PEP *itself*, not so much the API or implementation.

I think that said API and implementation is fine, but FAR too
narrowly scoped to claim to be "futures" or "execute computations
asynchronously", as the PEP calls it.  It's really just a nice task
queuing system.

Now, if the PEP were *scoped* as such, i.e., "hey, let's just have a
nice multithread/multiprocess task queuing implementation in the
stdlib", I would be +1.  It's a handy utility to have.

But I think that the scope given by the PEP appears overly ambitious
compared to what is actually being delivered; this seems less of a
"futures API" and more like a couple of utility functions for waiting
on threads and processes.

To rise to the level of an API, it seems to me that it would need to
address interop with coroutines and async frameworks, where the idea
of "futures" seems much more relevant than simple
synchronous-but-parallel scripts.  (It should also have better tools
for working with futures asynchronously, because, hey, it says right
there in the title, "execute computations asynchronously".)

Anyway, I'd like to see the answers to (at *least*) the following
issues fleshed out in the PEP, if you want it to really be a "futures
API", vs. "nice task queue in the stdlib":

* Compare/contrast alternatives now available
* Address the issue of competing event loops and sharing/passing
executors among code
* Either offer a way for executed code to re-enter its own executor
(e.g. via an optional parameter), or explain why this was considered
and rejected
* Address interoperability with coroutines and async frameworks, or
clearly explain why such is out of scope

(Personally, I think it would be better to just drop the ambitious
title and scope, and go for the "nice task queue" scope.  I imagine,
too, that in that case Jean-Paul wouldn't need to worry about it
being raised as a future objection to Deferreds or some such getting
into the stdlib.)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Gregory Ewing
Greg Ewing wrote:
> The thing to consider, I think, is whether it makes sense in
> a large proportion of use cases to ignore the fact that the
> cancel failed and carry on regardless. If not, then raising an
> exception makes it much harder to accidentally ignore the
> situation.

Cancelling operations is generally a best effort activity - having to
wrap it an exception handler would be annoying.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by PJ Eby
P.J. Eby wrote:
> (Personally, I think it would be better to just drop the ambitious title
> and scope, and go for the "nice task queue" scope.  I imagine, too, that
> in that case Jean-Paul wouldn't need to worry about it being raised as a
> future objection to Deferreds or some such getting into the stdlib.)

This may be a terminology thing - to me futures *are* just a nice way to
handle farming tasks out to worker threads or processes. You seem to see
them as something more comprehensive than that.

I agree the PEP should just target what the current implementation
provides and put whatever scope limitations are needed in the preamble
text to make that clear.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jesse Noller
In reply to this post by Dj Gilcrease
On Sat, Mar 6, 2010 at 10:09 PM, Dj Gilcrease <[hidden email]> wrote:

> After playing with the API for a while & running into many issues with
> the examples & tests crashing windows I decided to modify the API a
> little and fix up the examples so they dont crash windows based
> computers.
>
> http://code.google.com/p/pythonfutures/issues/detail?id=1
>
> API Change that changes the current Executor to ExecutorBase and
> adds a new Executor class that is used like
>
> futures.Executor() # creates an executor that uses threading and a
> max_workers = to the number of cpus
>
> futures.Executor(use='process') # Creates an executor that uses
> multiprocessing and a max_workers = to the number of cpus
>
> futures.Executor(max_workers=5) # threading again, just specifying the
> number of workers
>
> futures.Executor(use='process', max_workers=5) # back to multiprocessing,
> but with the max_workers specified

Making the tests and examples happy on windows is fine; but some
explanation is needed for the API changes.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
In reply to this post by Nick Coghlan
At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
>P.J. Eby wrote:
> > (Personally, I think it would be better to just drop the ambitious title
> > and scope, and go for the "nice task queue" scope.  I imagine, too, that
> > in that case Jean-Paul wouldn't need to worry about it being raised as a
> > future objection to Deferreds or some such getting into the stdlib.)
>
>This may be a terminology thing - to me futures *are* just a nice way to
>handle farming tasks out to worker threads or processes. You seem to see
>them as something more comprehensive than that.

Actual futures are, yes.  Specifically, futures are a mechanism for
asynchronous computation, whereas the PEP seems to be all about
synchronously managing parallel tasks.  That's a huge difference.

Technically, the things in the PEP (and by extension, Java's futures)
match the letter of the definition of a future, but not (IMO) the
spirit.  There's no clean way to compose them, and at base they're
more about parallelism than asynchrony.


>I agree the PEP should just target what the current implementation
>provides and put whatever scope limitations are needed in the preamble
>text to make that clear.

Yep.  I'm just saying "parallel task queueing" is a much better
description of what the implementation is/does, and would suggest
renaming Future -> Task and Executor -> WorkerPool or some
such.  These names would be *much* clearer to people who've never
heard of futures, as well as more appropriate to the actual scope of
what this does.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12345