[PEP 3148] futures - execute computations asynchronously

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan

On 10 Mar 2010, at 08:32, Dj Gilcrease wrote:

> On Mon, Mar 8, 2010 at 2:11 PM,  <[hidden email]> wrote:
>> Getting rid of the process-global state like this simplifies  
>> testing (both
>> testing of the executors themselves and of application code which  
>> uses
>> them).  It also eliminates the unpleasant interpreter shutdown/module
>> globals interactions that have plagued a number of stdlib systems  
>> that keep
>> global state.
>
>
> Ok the new patch is submitted @
> http://code.google.com/p/pythonfutures/issues/detail?id=1

Cool, thanks.

> *note there are 2 tests that fail and 1 test that dead locks on
> windows even without this patch, the deadlock test I am skipping in
> the patch and the two that fail do so for a reason that does not make
> sense to me.

I'll investigate but I don't have convenient access to a windows  
machine.

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Gregory Ewing

On 9 Mar 2010, at 08:39, Greg Ewing wrote:

> Terry Reedy wrote:
>> Looking more close, I gather that the prime results will be printed  
>> 'in order' (waiting on each even if others are done) while the url  
>> results will be printed 'as available'.
>
> Seems to me that if you care about the order of the results,
> you should be able to just wait for each result separately
> in the order you want them. Something like
>
>  task1 = start_task(proc1)
>  task2 = start_task(proc2)
>  task3 = start_task(proc3)
>  result1 = task1.wait_for_result()
>  result2 = task2.wait_for_result()
>  result3 = task3.wait_for_result()

You can write this as:

executor = ...
future1 = executor.submit(proc1)
future2 = executor.submit(proc2)
future3 = executor.submit(proc3)
result1 = task1.result()
result2 = task2.result()
result3 = task3.result()

> This would also be a natural way to write things even if
> you don't care about the order, but you need all the results
> before proceeding. You're going to be held up until the
> longest-running task completes anyway, so it doesn't matter
> if some of them finish earlier and have to sit around
> waiting for you to collect the result.

Often you don't want to continue if there is a failure.

In the example that you gave, if "proc3" raises an exception  
immediately, you still wait for "proc1" and "proc2" to complete even  
though you will end up discarding their results.

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by exarkun

On 9 Mar 2010, at 08:11, [hidden email] wrote:

> On 08:56 pm, [hidden email] wrote:
>> On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease  
>> <[hidden email]> wrote:
>>> A style I have used in my own code in the past is a Singleton class
>>> with register and create methods, where the register takes a
>>> name(string) and the class and the create method takes the name and
>>> *args, **kwargs and acts as a factory.
>>
>>
>> So I decided to play with this design a little and since I made it a
>> singleton I decided to place all the thread/process tracking and exit
>> handle code in it instead of having the odd semi-global scoped
>> _shutdown, _thread_references, _remove_dead_thread_references and
>> _python_exit objects floating around in each executor file, seems to
>> work well. The API would be
>>
>> from concurrent.futures import executors
>>
>> executor = executors.create(NAME, *args, **kwargs) # NAME is  
>> 'process'
>> or 'thread' by default
>>
>>
>> To create your own executor you create your executor class and add  
>> the
>> following at the end
>
> Getting rid of the process-global state like this simplifies testing  
> (both testing of the executors themselves and of application code  
> which uses them).  It also eliminates the unpleasant interpreter  
> shutdown/module globals interactions that have plagued a number of  
> stdlib systems that keep global state.

I'm not sure what you mean, could you clarify?

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Terry Reedy

On 9 Mar 2010, at 03:21, Terry Reedy wrote:

> On 3/6/2010 4:20 AM, Brian Quinlan wrote:
>>
>> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:
>>
>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>    import futures
>>>
>>>
>>> +1 on the idea, -1 on the name. It's too similar to "from __future__
>>> import ...".
>>>
>>> Also, the PEP should probably link to the discussions on stdlib-sig?
>>
>> I thoug ht about that but this discussion is spread over many threads
>> and many months.
>
> This is pretty typical. I would say just that, and link to the first.
> "This PEP was discussed over many months in many threads in the  
> stdlib-sig list. The first was .... . Python-dev discussion occured  
> in <this thread>.

I'll add that.

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Brian Quinlan
Brian Quinlan wrote:
>> Getting rid of the process-global state like this simplifies testing
>> (both testing of the executors themselves and of application code
>> which uses them).  It also eliminates the unpleasant interpreter
>> shutdown/module globals interactions that have plagued a number of
>> stdlib systems that keep global state.
>
> I'm not sure what you mean, could you clarify?

Assuming your question refers to the second sentence, Jean-Paul is
referring to a trick of the CPython interpreter when it terminates. To
maximise the chances of objects being deleted properly rather than just
dumped from memory when the process exits, module dictionaries are
filled with None values before the interpreter shuts down.

This can cause weirdness (usually intermittent name errors during
shutdown) when __del__ methods directly or indirectly reference module
globals.

One of the easiest ways to avoid that is to put the state on a singleton
object, then give the affected classes a reference to that object.

Cheers,
Nick.

P.S. This problem is actually the reason we don't have a context manager
for temporary directories yet. Something that should have been simple
became a twisty journey down the rabbit hole:
http://bugs.python.org/issue5178

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
In reply to this post by Skip Montanaro-3
[hidden email] wrote:
>     >> I'm +1 on adding a nice task queuing system, -1 on calling it by any
>     >> other name.  ;-)
>
>     Nick> As Guido said, let's call the nice task queuing system "futures"
>     Nick> and point people wanting a full-power asynchronous process model
>     Nick> to Twisted
>
> Can this module at least be pushed down into a package?  I think
> "concurrent" or "concurrency" were both suggested at one point.

Yep, I believe "concurrent.futures" was picked as the name elsewhere in
the thread.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Nick Coghlan

On 10 Mar 2010, at 23:32, Nick Coghlan wrote:

> Brian Quinlan wrote:
>>> Getting rid of the process-global state like this simplifies testing
>>> (both testing of the executors themselves and of application code
>>> which uses them).  It also eliminates the unpleasant interpreter
>>> shutdown/module globals interactions that have plagued a number of
>>> stdlib systems that keep global state.
>>
>> I'm not sure what you mean, could you clarify?
>
> Assuming your question refers to the second sentence, Jean-Paul is
> referring to a trick of the CPython interpreter when it terminates. To
> maximise the chances of objects being deleted properly rather than  
> just
> dumped from memory when the process exits, module dictionaries are
> filled with None values before the interpreter shuts down.
>
> This can cause weirdness (usually intermittent name errors during
> shutdown) when __del__ methods directly or indirectly reference module
> globals.


Ah. I'm familiar with this problem. My approach was to install an exit  
handler that ensures that all pending futures are complete and all  
threads and processes exit before allowing the interpreter to exit.

Cheers,
Brian

> One of the easiest ways to avoid that is to put the state on a  
> singleton
> object, then give the affected classes a reference to that object.
>
> Cheers,
> Nick.
>
> P.S. This problem is actually the reason we don't have a context  
> manager
> for temporary directories yet. Something that should have been simple
> became a twisty journey down the rabbit hole:
> http://bugs.python.org/issue5178
>
> --
> Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
In reply to this post by Nick Coghlan
Nick Coghlan wrote:
 > You may want to consider providing global thread and process  
executors
 > in the futures module itself. Code which just wants to say "do this  
in
 > the background" without having to manage the lifecycle of its own
 > executor instance is then free to do so. I've had a lot of experience
 > with a framework that provides this and it is *very* convenient (it's
 > also a good way to avoid deadlocks due to synchronous notification  
APIs).

This seems like a reasonable idea to me.

I take it that the thread/process pool should be unlimited in size.  
Should every thread/process exit when it finishes its job or should  
there be a smarter collection strategy?

Cheers,
Brian
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Nick Coghlan
Brian Quinlan wrote:
> I take it that the thread/process pool should be unlimited in size.
> Should every thread/process exit when it finishes its job or should
> there be a smarter collection strategy?

I'd be inclined to do something slightly smarter, similar to what we do
with memory overallocation for mutable containers.

Cheers,
Nick.

--
Nick Coghlan   |   [hidden email]   |   Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [PEP 3148] futures - execute computations asynchronously

Brian Quinlan
I've updated the PEP to include:
- completion callbacks (for interoperability with Twisted Deferreds)
- a pointer to the discussion on stdlig-sig

See:
http://svn.python.org/view/peps/trunk/pep-3148.txt?r1=78618&r2=80679

Rejected ideas:
- Having a registration system for executors

Not yet addressed:
- where the package should live (someone in a "concurrent" package seems fine)
- having global executors with unbounded worker counts as a convenience [1]

[1] There are a few issues with global executors that need to be thought thought through i.e. when should workers be created and when should they be terminated. I'd be happy to defer this idea unless someone is passionate about it (in which case it would be great if they'd step in with concrete ideas).

Cheers,
Brian

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40nabble.com
12345