Parallel IPython: Spawn Tasks from Tasks, direct scheduling

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel IPython: Spawn Tasks from Tasks, direct scheduling

Andreas Buhr
Dear IPython Community,

I have a hard time understanding the task mechanisms in parallel
IPython, maybe you can help me please?

I'd like to have lots of data lying on the workers, i.e. Python objects.
Then I'd like to call member functions of these objects, using tasks. So
I would like to have a task which says: "Go to worker A and there
execute method B on object C (which is already there)".

Is this possible with IPython?

thanks a lot in advance and best regards,
Andreas
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Parallel IPython: Spawn Tasks from Tasks, direct scheduling

MinRK

There are two ways to do this:

  1. use parallel.Reference for the task function and/or any arguments
  2. use lambdas and rely on globals to fetch names from the interactive namespace

A notebook illustrating both approaches

-MinRK


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Parallel IPython: Spawn Tasks from Tasks, direct scheduling

Andreas Buhr
Hi,

wow, thanks a lot for your help, that's great.

On 10/17/2013 09:06 PM, MinRK wrote:
> A notebook illustrating both approaches
> <http://nbviewer.ipython.org/7030408>

>From inside of those member functions, can I spawn new tasks? Can I just
get a new 'Client()', even when I am on a worker?

And if I spawn a new task on a worker, and that new task is to be
executed on the exact same machine, is there a way to execute that task
without going back to the HUB/scheduler?

thanks a lot again,
best regards,
Andreas

_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Parallel IPython: Spawn Tasks from Tasks, direct scheduling

MinRK

On Thu, Oct 17, 2013 at 12:16 PM, Andreas Buhr <[hidden email]> wrote:
Hi,

wow, thanks a lot for your help, that's great.

On 10/17/2013 09:06 PM, MinRK wrote:
> A notebook illustrating both approaches
> <http://nbviewer.ipython.org/7030408>

>From inside of those member functions, can I spawn new tasks? Can I just
get a new 'Client()', even when I am on a worker?

You can create a client and submit new tasks from there,
though there can be hazards associated with that in terms of the number of sockets created.
 

And if I spawn a new task on a worker, and that new task is to be
executed on the exact same machine, is there a way to execute that task
without going back to the HUB/scheduler?

If you spawn new tasks, they are submitted in exactly the same way as all other tasks via the scheduler. If you want it to run on the same machine that's submitting the task, why not just call the function directly?

-MinRK
 

thanks a lot again,
best regards,
Andreas

_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Parallel IPython: Spawn Tasks from Tasks, direct scheduling

Andreas Buhr
Hi MinRK,

On 10/17/2013 09:20 PM, MinRK wrote:
>
>     And if I spawn a new task on a worker, and that new task is to be
>     executed on the exact same machine, is there a way to execute that task
>     without going back to the HUB/scheduler?
>
>
> If you spawn new tasks, they are submitted in exactly the same way as
> all other tasks via the scheduler. If you want it to run on the same
> machine that's submitting the task, why not just call the function directly?

Because I would like to have also task priorities and there might be
tasks with higher priority waiting. Are there task priorities in IPython?
Also because I would not like the task spawner to know where the task
will be executed. At the end, I'd like to say "Call function A on object
B with arguments C", and the system I am about to create should carry
the task to B's home and execute it there. And it should not go over the
wire if B happen to live on the same machine.

Thanks again for your help :-)

best regards,
Andreas

_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: Parallel IPython: Spawn Tasks from Tasks, direct scheduling

MinRK
That's interesting.  If you are doing engine assignment yourself, I would probably say that you should have that logic at a level slightly higher than IPython.  Then on task submission, you can call out to a function directly if your code determines that it should be run locally rather than submitted via IPython.


On Thu, Oct 17, 2013 at 12:34 PM, Andreas Buhr <[hidden email]> wrote:
Hi MinRK,

On 10/17/2013 09:20 PM, MinRK wrote:
>
>     And if I spawn a new task on a worker, and that new task is to be
>     executed on the exact same machine, is there a way to execute that task
>     without going back to the HUB/scheduler?
>
>
> If you spawn new tasks, they are submitted in exactly the same way as
> all other tasks via the scheduler. If you want it to run on the same
> machine that's submitting the task, why not just call the function directly?

Because I would like to have also task priorities and there might be
tasks with higher priority waiting. Are there task priorities in IPython?
Also because I would not like the task spawner to know where the task
will be executed. At the end, I'd like to say "Call function A on object
B with arguments C", and the system I am about to create should carry
the task to B's home and execute it there. And it should not go over the
wire if B happen to live on the same machine.

Thanks again for your help :-)

best regards,
Andreas

_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev