[PEP 3148] futures - execute computations asynchronously

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jeffrey Yasskin-2
On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby <[hidden email]> wrote:

> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
>>
>> P.J. Eby wrote:
>> > (Personally, I think it would be better to just drop the ambitious title
>> > and scope, and go for the "nice task queue" scope.  I imagine, too, that
>> > in that case Jean-Paul wouldn't need to worry about it being raised as a
>> > future objection to Deferreds or some such getting into the stdlib.)
>>
>> This may be a terminology thing - to me futures *are* just a nice way to
>> handle farming tasks out to worker threads or processes. You seem to see
>> them as something more comprehensive than that.
>
> Actual futures are, yes.  Specifically, futures are a mechanism for
> asynchronous computation, whereas the PEP seems to be all about
> synchronously managing parallel tasks.  That's a huge difference.
>
> Technically, the things in the PEP (and by extension, Java's futures) match
> the letter of the definition of a future, but not (IMO) the spirit.  There's
> no clean way to compose them, and at base they're more about parallelism
> than asynchrony.

Do you have an example of a language or library that uses the term
"future" to refer to what you're talking about? I'm curious to see
what it looks like.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:

>On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby <[hidden email]> wrote:
> > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
> >>
> >> P.J. Eby wrote:
> >> > (Personally, I think it would be better to just drop the ambitious title
> >> > and scope, and go for the "nice task queue" scope. Â I
> imagine, too, that
> >> > in that case Jean-Paul wouldn't need to worry about it being raised as a
> >> > future objection to Deferreds or some such getting into the stdlib.)
> >>
> >> This may be a terminology thing - to me futures *are* just a nice way to
> >> handle farming tasks out to worker threads or processes. You seem to see
> >> them as something more comprehensive than that.
> >
> > Actual futures are, yes. Â Specifically, futures are a mechanism for
> > asynchronous computation, whereas the PEP seems to be all about
> > synchronously managing parallel tasks. Â That's a huge difference.
> >
> > Technically, the things in the PEP (and by extension, Java's futures) match
> > the letter of the definition of a future, but not (IMO) the
> spirit. Â There's
> > no clean way to compose them, and at base they're more about parallelism
> > than asynchrony.
>
>Do you have an example of a language or library that uses the term
>"future" to refer to what you're talking about? I'm curious to see
>what it looks like.

The wikipedia page menetioned earlier in the discussion has several
examples.  Twisted's Deferreds are probably the best example of such
a system in Python, though, as far as I know.  The features proposed
in the PEP are basically akin to Twisted's deferToThread(), i.e.
"return a future for running this code in some other thread".

However, you could easily make your own thread or process pool for
executing tasks, using Deferreds as your "future" objects;
deferToThread() is not particularly special, it's just one possible
"executor", to use the PEP's terminology.  Likewise, deferreds are
not limited to representing parallel task execution - they can also
represent asynchronous events such as incoming I/O or a waited-for mouse click.

Deferreds allow composing calculation by stacking callbacks (in
continuation-passing style), but they're most Pythonic (IMO) when
used with a coroutine wrapping system like Eventlet or my earlier
peak.events framework, so that you can use them in "yield"
expressions.  That way you can write synchronous-looking code that
nonetheless executes asynchronously.

Twisted uses Deferreds extensively, both for I/O and IPC, and of
course for database access.  However, the 'defer' module itself is
conceptually distinct from both the I/O system, event loop, and
threads: it's a largely-generic framework for managing asynchronous
computation.  Parallel task queueing is simply *one* possible
application of Deferreds; their actual purpose is to provide a common
API for working with values that aren't yet known.

That is to say, futures.

In contrast, the PEP manages only a very limited sort of "future" -
the completion of a single task, which must be synchronously waited
for.  If one "future" in the PEP needs to wait for another, you tie
up that thread or process in a busy loop...  which rather limits how
much parallelism you can have.

Deferreds also allow you to write code that doesn't *care* whether
the value has been determined or not...  you can simply stack on what
code you want to have run when the value is available, and it runs
immediately if the value is there.

Likewise, a deferred is not tied to the execution of a single piece
of code - it's more of a rendezvous or synchronization point, that
can be given a value to "fire" with.  (In contrast, the PEP's
"futures" are tightly coupled to a task execution model: they have a
notion of running and being cancelled, for example.)

For these reasons, I'd say that Deferreds are much more deserving of
the title "futures", especially since there is no need to call a
simple task queue a "futures" system.  (Unless you're Java, I
suppose, in which case you might want the name for marketing
reasons.  ;-) But in Python I assume we'd like to call a spade a
spade, and a task queue a task queue.)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

R. David Murray
In reply to this post by PJ Eby
On Sun, 07 Mar 2010 10:48:09 -0500, "P.J. Eby" <[hidden email]> wrote:

> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
> >I agree the PEP should just target what the current implementation
> >provides and put whatever scope limitations are needed in the preamble
> >text to make that clear.
>
> Yep.  I'm just saying "parallel task queueing" is a much better
> description of what the implementation is/does, and would suggest
> renaming Future -> Task and Executor -> WorkerPool or some
> such.  These names would be *much* clearer to people who've never
> heard of futures, as well as more appropriate to the actual scope of
> what this does.

For what it's worth: I don't have any particular knowledge in this area.
I did loosely follow the stdlib-sig discussion.  I wasn't really sure
exactly what the module was about or what a 'future' was, or why I would
want to use one.  I did get that it was about parallel execution of tasks,
but it seemed like there had to be more to it than that.  Hearing it
called a 'worker pool' makes a lightbulb go off and I can now understand
why this would be a useful facility to have in the standard library.

--
R. David Murray                                      www.bitdance.com
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Jeffrey Yasskin-2
In reply to this post by PJ Eby
On Sun, Mar 7, 2010 at 9:57 AM, P.J. Eby <[hidden email]> wrote:

> At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>> Do you have an example of a language or library that uses the term
>> "future" to refer to what you're talking about? I'm curious to see
>> what it looks like.
>
> The wikipedia page menetioned earlier in the discussion has several
> examples.  Twisted's Deferreds are probably the best example of such a
> system in Python, though, as far as I know.  The features proposed in the
> PEP are basically akin to Twisted's deferToThread(), i.e. "return a future
> for running this code in some other thread".

Nearly all of the wikipedia page's examples refer to exactly this kind
of system, but with syntax to support it instead of putting it in a
library. The distinction between futures and deferreds is that futures
are designed to block, while deferreds are designed to continue
through callbacks. We could say that futures are to deferreds as
threading.Lock is to mutex.mutex.

> That way you
> can write synchronous-looking code that nonetheless executes asynchronously.

 "Asynchronous" in the PEP refers to how the task behind the future
runs, not how users wait for the future. Is there a better way to say
that?

> Twisted uses Deferreds extensively, both for I/O and IPC, and of course for
> database access.  However, the 'defer' module itself is conceptually
> distinct from both the I/O system, event loop, and threads: it's a
> largely-generic framework for managing asynchronous computation.  Parallel
> task queueing is simply *one* possible application of Deferreds; their
> actual purpose is to provide a common API for working with values that
> aren't yet known.
>
> That is to say, futures.
>
> In contrast, the PEP manages only a very limited sort of "future" - the
> completion of a single task, which must be synchronously waited for.  If one
> "future" in the PEP needs to wait for another, you tie up that thread or
> process in a busy loop...  which rather limits how much parallelism you can
> have.

So is it that you just don't like the idea of blocking, and want to
stop anything that relies on it from getting into the standard
library?

Given the set_result and set_exception methods, it's pretty
straightforward to fill in the value of a future from something that
isn't purely computational. Given a way to register "on-done"
callbacks with the future, it would be straightforward to wait for a
future without blocking, too.

Jeffrey
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

PJ Eby
At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>So is it that you just don't like the idea of blocking, and want to
>stop anything that relies on it from getting into the standard library?

Um, no.  As I said before, call it a "parallel task queue" or
"parallel task manager" or something to that general effect and I'm on board.

It may not be in the Zen of Python, but ISTM that names should
generally follow use cases.  It is something of a corollary to "one
obvious way to do it", in that if you see something whose name
matches what you want to do, then it should be obvious that that's
the way in question.  ;-)

The use cases for "parallel task queues", however, are a subset of
those for "futures" in the general case.  Since the proposed module
addresses most of the former but very little of the latter, calling
it futures is inappropriate.

Specifically, it's:

1. Confusing to people who don't know what futures are (see e.g R.D.
Murray's post), and

2. Underpowered for people who expect/want a more fully-featured
futures system along the lines of E or Deferreds.

It seems that the only people for whom it's an intuitively correct
description are people who've only had experience with more limited
futures models (like Java's).  However, these people should not have
a problem understanding the notion of parallel task queueing or task
management, so changing the name isn't really a loss for them, and
it's a gain for everybody else.


>  Given the set_result and set_exception methods, it's pretty
> straightforward to fill in the value of a future from something
> that isn't purely computational.

Those are described as "internal" methods in the PEP; by contrast,
the Deferred equivalents are part of the public API.


>  Given a way to register "on-done" callbacks with the future, it
> would be straightforward to wait for a future without blocking, too.

Yes, and with a few more additions besides that one, you might be on
the way to an actual competitor for Deferreds.  For example: retry
support, chaining, logging, API for transparent result processing,
coroutine support, co-ordination tools like locks, sempaphores and queues, etc.

These are all things you would very likely want or need if you
actually wanted to write a program using futures as *your main
computational model*, vs. just needing to toss out some parallel
tasks in a primarily synchronous program.

Of course, Deferreds are indeed overkill if all you're ever going to
want is a few parallel tasks, unless you're already skilled in using
Twisted or some wrapper for it.

So, I totally support having a simple task queue in the stdlib, as
there are definitely times I would've used such a thing for a quick
script, if it were available.

However, I've *also* had use cases for using futures as a
computational model, and so that's what I originally thought this PEP
was about.  After the use cases were clarified, though, it seems to
me that *calling* it futures is a bad idea, because it's really just
a nice task queuing system.

I'm +1 on adding a nice task queuing system, -1 on calling it by any
other name.  ;-)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by Jesse Noller
On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller <[hidden email]> wrote:
> Making the tests and examples happy on windows is fine; but some
> explanation is needed for the API changes.
>

My primary motivation behind the API change is so there is just a
single public Executor class that you tell what system to use instead
of two separate classes. The use case I was thinking about is when a
user is is unsure which system (threads or processes) they want to use
so they just build the system with the defaults (which is threads)
then it is a little easier to switch it to processes in the future
instead of having to change imports and all instances of the class you
just change the use keyword to switch between systems
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Guido van Rossum
In reply to this post by PJ Eby
On Sun, Mar 7, 2010 at 11:56 AM, P.J. Eby <[hidden email]> wrote:

> At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>>
>> So is it that you just don't like the idea of blocking, and want to stop
>> anything that relies on it from getting into the standard library?
>
> Um, no.  As I said before, call it a "parallel task queue" or "parallel task
> manager" or something to that general effect and I'm on board.
>
> It may not be in the Zen of Python, but ISTM that names should generally
> follow use cases.  It is something of a corollary to "one obvious way to do
> it", in that if you see something whose name matches what you want to do,
> then it should be obvious that that's the way in question.  ;-)
>
> The use cases for "parallel task queues", however, are a subset of those for
> "futures" in the general case.  Since the proposed module addresses most of
> the former but very little of the latter, calling it futures is
> inappropriate.
>
> Specifically, it's:
>
> 1. Confusing to people who don't know what futures are (see e.g R.D.
> Murray's post), and

This is not a problem. We will document what we consider a future.

> 2. Underpowered for people who expect/want a more fully-featured futures
> system along the lines of E or Deferreds.

This sounds like an underhanded slur towards the PEP.

> It seems that the only people for whom it's an intuitively correct
> description are people who've only had experience with more limited futures
> models (like Java's).  However, these people should not have a problem
> understanding the notion of parallel task queueing or task management, so
> changing the name isn't really a loss for them, and it's a gain for
> everybody else.

I expect that the majority of Python users fall either in camp #1
(never heard of futures, will be happy to learn about what Python
calls futures) or camp #3 (have used Java futures). The users of E can
be counted on a few hands. Deferreds are used heavily in some Python
circles but most Python users (myself included) have at most a very
vague idea of them. Also, as you clarify below, Deferreds are so much
more powerful that they can't possibly be mistaken for futures (as
defined by this PEP). Plus they already have a name.

>>  Given the set_result and set_exception methods, it's pretty
>> straightforward to fill in the value of a future from something that isn't
>> purely computational.
>
> Those are described as "internal" methods in the PEP; by contrast, the
> Deferred equivalents are part of the public API.
>
>
>>  Given a way to register "on-done" callbacks with the future, it would be
>> straightforward to wait for a future without blocking, too.
>
> Yes, and with a few more additions besides that one, you might be on the way
> to an actual competitor for Deferreds.  For example: retry support,
> chaining, logging, API for transparent result processing, coroutine support,
> co-ordination tools like locks, sempaphores and queues, etc.
>
> These are all things you would very likely want or need if you actually
> wanted to write a program using futures as *your main computational model*,
> vs. just needing to toss out some parallel tasks in a primarily synchronous
> program.
>
> Of course, Deferreds are indeed overkill if all you're ever going to want is
> a few parallel tasks, unless you're already skilled in using Twisted or some
> wrapper for it.
>
> So, I totally support having a simple task queue in the stdlib, as there are
> definitely times I would've used such a thing for a quick script, if it were
> available.
>
> However, I've *also* had use cases for using futures as a computational
> model, and so that's what I originally thought this PEP was about.  After
> the use cases were clarified, though, it seems to me that *calling* it
> futures is a bad idea, because it's really just a nice task queuing system.
>
> I'm +1 on adding a nice task queuing system, -1 on calling it by any other
> name.  ;-)

So let's focus on the functionality of the task queuing system, and
stick to roughly the functionality proposed in the PEP.

The name is a non-issue and further discussion ought to be sent to
[hidden email].

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by PJ Eby
P.J. Eby wrote:
> I'm +1 on adding a nice task queuing system, -1 on calling it by any
> other name.  ;-)

As Guido said, let's call the nice task queuing system "futures" and
point people wanting a full-power asynchronous process model to Twisted
- while the Deferred API may technically be independent of the rest of
the framework, you need at least some of the other tools for
asynchronous operations to make it really shine.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Dj Gilcrease
Dj Gilcrease wrote:

> On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller <[hidden email]> wrote:
>> Making the tests and examples happy on windows is fine; but some
>> explanation is needed for the API changes.
>>
>
> My primary motivation behind the API change is so there is just a
> single public Executor class that you tell what system to use instead
> of two separate classes. The use case I was thinking about is when a
> user is is unsure which system (threads or processes) they want to use
> so they just build the system with the defaults (which is threads)
> then it is a little easier to switch it to processes in the future
> instead of having to change imports and all instances of the class you
> just change the use keyword to switch between systems

Wouldn't a factory function serve that purpose just as well? Or even
just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor".

That last form has the virtue that you can retrieve your executor from
anywhere rather than being limited to the two provided by the
concurrent.futures model.

I think the string based approach actually unduly constrains the API
despite superficially appearing to make it more flexible.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Terry Reedy
In reply to this post by Nick Coghlan
On 3/8/2010 6:14 AM, Nick Coghlan wrote:
> P.J. Eby wrote:
>> I'm +1 on adding a nice task queuing system, -1 on calling it by any
>> other name.  ;-)
>
> As Guido said, let's call the nice task queuing system "futures" and

I was confused by 'futures' also until Philip explained it as task-queue
or task-pool, and hence also do not like it.

Since the examples in the PEP do *NOT* give example output, it was not
clear to me whether execution or the termination thereof is ordered
(queue) or not (pool). Looking more close, I gather that the prime
results will be printed 'in order' (waiting on each even if others are
done) while the url results will be printed 'as available'. Adding 'will
print ...' and 'might print ...' outputs would help.

> point people wanting a full-power asynchronous process model to Twisted

That could be done in the PEP to clarify its scope.

Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Terry Reedy
In reply to this post by Brian Quinlan
On 3/6/2010 4:20 AM, Brian Quinlan wrote:

>
> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:
>
>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     import futures
>>
>>
>> +1 on the idea, -1 on the name. It's too similar to "from __future__
>> import ...".
>>
>> Also, the PEP should probably link to the discussions on stdlib-sig?
>
> I thoug ht about that but this discussion is spread over many threads
> and many months.

This is pretty typical. I would say just that, and link to the first.
"This PEP was discussed over many months in many threads in the
stdlib-sig list. The first was .... . Python-dev discussion occured in
<this thread>.

tjr



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by Nick Coghlan
On Mon, Mar 8, 2010 at 4:25 AM, Nick Coghlan <[hidden email]> wrote:
> Wouldn't a factory function serve that purpose just as well? Or even
> just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor".
>
> That last form has the virtue that you can retrieve your executor from
> anywhere rather than being limited to the two provided by the
> concurrent.futures model.
>
> I think the string based approach actually unduly constrains the API
> despite superficially appearing to make it more flexible.

mm you are correct, I went with the string approach because I was
experimenting with 3 additional executor types and wanted to be able
to switch between or intermix them them without having to change
imports and didnt feel like writing a register class with a factory
method.

A style I have used in my own code in the past is a Singleton class
with register and create methods, where the register takes a
name(string) and the class and the create method takes the name and
*args, **kwargs and acts as a factory.

Would this style be better or would it be better to just leave it with
the two executor classes? I tend to dislike multiple classes for a
what is essentially a Strategy of a concept and factories are
something I tend to forget about until well after my initial idea has
formed into a proof of concept.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Skip Montanaro-3
In reply to this post by Nick Coghlan

    >> I'm +1 on adding a nice task queuing system, -1 on calling it by any
    >> other name.  ;-)

    Nick> As Guido said, let's call the nice task queuing system "futures"
    Nick> and point people wanting a full-power asynchronous process model
    Nick> to Twisted

Can this module at least be pushed down into a package?  I think
"concurrent" or "concurrency" were both suggested at one point.

Skip
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by Dj Gilcrease
On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease <[hidden email]> wrote:
> A style I have used in my own code in the past is a Singleton class
> with register and create methods, where the register takes a
> name(string) and the class and the create method takes the name and
> *args, **kwargs and acts as a factory.


So I decided to play with this design a little and since I made it a
singleton I decided to place all the thread/process tracking and exit
handle code in it instead of having the odd semi-global scoped
_shutdown, _thread_references, _remove_dead_thread_references and
_python_exit objects floating around in each executor file, seems to
work well. The API would be

from concurrent.futures import executors

executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
or 'thread' by default


To create your own executor you create your executor class and add the
following at the end

from concurrent.futures import executors, ExecutorBase
class MyExecutor(ExecutorBase): ...
executors.register(NAME, MyExecutor)

It checks to see if your executor is a subclass of ExecutorBase, but
only does a UserWarning if it is not since you should know what
methods are required to be an executor like object, so if you are not
subclassing ExecutorBase you should suppress the UserWarning before
you register you class and un-suppress it after


Some Helper Methods/Properties on the executors Singleton
add_joinable_ref - This replaces the _thread_references.add system of
tracking threads, and it allows for adding processes as well hence the
joinable_ref name. It does check to make sure the ref being passes has
a join method then creates a weakref to it and adds it to a set. For
every thread or process your executor creates you should call this
with so it will be tracked properly

cleanup_joinable_refs - This replaces the
_remove_dead_thread_references that had to be written for each
executor individually. This should be called periodically, currently
it is only called when you create a new executor since it is a
blocking method (it uses a thread lock to make sure the set of
references does not change while it is discarding old ones)

shutdown - is a readonly property and replaces the _shutdown global
var that had to be created for each executor individually, it is set
in the executors destructor

__del__ - replaces the _python_exit method that had to be written for
each executor individually



If this API change isnt accepted its no big deal, since I am only
changing it due to personal pet peeves and the only real issue I had
with the package was scoping which has already been addressed by
adding it to a concurrent package
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

exarkun
On 08:56 pm, [hidden email] wrote:

>On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease <[hidden email]>
>wrote:
>>A style I have used in my own code in the past is a Singleton class
>>with register and create methods, where the register takes a
>>name(string) and the class and the create method takes the name and
>>*args, **kwargs and acts as a factory.
>
>
>So I decided to play with this design a little and since I made it a
>singleton I decided to place all the thread/process tracking and exit
>handle code in it instead of having the odd semi-global scoped
>_shutdown, _thread_references, _remove_dead_thread_references and
>_python_exit objects floating around in each executor file, seems to
>work well. The API would be
>
>from concurrent.futures import executors
>
>executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
>or 'thread' by default
>
>
>To create your own executor you create your executor class and add the
>following at the end

Getting rid of the process-global state like this simplifies testing
(both testing of the executors themselves and of application code which
uses them).  It also eliminates the unpleasant interpreter
shutdown/module globals interactions that have plagued a number of
stdlib systems that keep global state.

Jean-Paul
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by Terry Reedy
Terry Reedy wrote:
> Looking more close, I gather that the prime
> results will be printed 'in order' (waiting on each even if others are
> done) while the url results will be printed 'as available'.

Seems to me that if you care about the order of the results,
you should be able to just wait for each result separately
in the order you want them. Something like

   task1 = start_task(proc1)
   task2 = start_task(proc2)
   task3 = start_task(proc3)
   result1 = task1.wait_for_result()
   result2 = task2.wait_for_result()
   result3 = task3.wait_for_result()

This would also be a natural way to write things even if
you don't care about the order, but you need all the results
before proceeding. You're going to be held up until the
longest-running task completes anyway, so it doesn't matter
if some of them finish earlier and have to sit around
waiting for you to collect the result.

--
Greg

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Gregory Ewing
In reply to this post by Dj Gilcrease
Dj Gilcrease wrote:

> executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
> or 'thread' by default
>
> from concurrent.futures import executors, ExecutorBase
> class MyExecutor(ExecutorBase): ...
> executors.register(NAME, MyExecutor)

I don't understand the reason for using a registration system
rather than just importing names from a module.

You mentioned wanting to globally change the executor class
being used by a program without having to make changes throughout.
Registering a different class under the same name would be one
way to do that, but you could achieve the same thing just by
assigning to a name in a module.

In other words, instead of inventing your own mechanism for
managing a namespace, just use a module as your namespace.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Antoine Pitrou
In reply to this post by exarkun
Le Mon, 08 Mar 2010 21:11:45 -0000,
[hidden email] a écrit :
>
> Getting rid of the process-global state like this simplifies testing
> (both testing of the executors themselves and of application code
> which uses them).  It also eliminates the unpleasant interpreter
> shutdown/module globals interactions that have plagued a number of
> stdlib systems that keep global state.

+1.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Terry Reedy
In reply to this post by Gregory Ewing
On 3/8/2010 4:39 PM, Greg Ewing wrote:

> Terry Reedy wrote:
>> Looking more close, I gather that the prime results will be printed
>> 'in order' (waiting on each even if others are done) while the url
>> results will be printed 'as available'.
>
> Seems to me that if you care about the order of the results,
> you should be able to just wait for each result separately
> in the order you want them. Something like
>
> task1 = start_task(proc1)
> task2 = start_task(proc2)
> task3 = start_task(proc3)
> result1 = task1.wait_for_result()
> result2 = task2.wait_for_result()
> result3 = task3.wait_for_result()

*If* I understand the first example correctly, this is effectively what
the first example does with two loops. But I was hoping for
clarification and amplification in the PEP.
>
> This would also be a natural way to write things even if
> you don't care about the order, but you need all the results
> before proceeding. You're going to be held up until the
> longest-running task completes anyway, so it doesn't matter
> if some of them finish earlier and have to sit around
> waiting for you to collect the result.
>


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Dj Gilcrease
In reply to this post by exarkun
On Mon, Mar 8, 2010 at 2:11 PM,  <[hidden email]> wrote:
> Getting rid of the process-global state like this simplifies testing (both
> testing of the executors themselves and of application code which uses
> them).  It also eliminates the unpleasant interpreter shutdown/module
> globals interactions that have plagued a number of stdlib systems that keep
> global state.


Ok the new patch is submitted @
http://code.google.com/p/pythonfutures/issues/detail?id=1

*note there are 2 tests that fail and 1 test that dead locks on
windows even without this patch, the deadlock test I am skipping in
the patch and the two that fail do so for a reason that does not make
sense to me.
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12345