[IPython-User] IPython parallel vs "standard" grid computing model

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[IPython-User] IPython parallel vs "standard" grid computing model

Tiago Antão
Dear all,

I have been a long standing user of ipython and ipython notebook. Most of my code is run on a grid (SGE mostly, but not only - LSF and Torque sometimes).

I was using my own ad-hoc parallelization code, but decided try to change the code to IPython parallel. And I running into what might be some conceptual dissonance between the standard grid model and my perception of IPython parallel.

So, in a standard grid application (at least my view of it), one submits jobs to a queue which will eventually be executed and terminate. The process life-cycle starts when the computation starts and ends when the computation ends.

It seems to me that IPython parallel (newbie here, I am guessing that I am wrong...) imposes that the engines have to be up on each cluster node, waiting for computation (even in load balancing mode): They are always on, sometimes computing, sometimes just sitting idle waiting for computation. My question is: is it possible to have a computing model that is based on "standard" grid queues, where an engine is queued for computation, does the required computation and dies? For me, asking cluster managers to allow a different computing model is not a possible avenue...

I apologize if this is a dead obvious question...

Tiago

--
"The truth may be out there, but the lies are already in your head" - Terry Pratchett

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel vs "standard" grid computing model

Martijn Vermaat
Hi Tiago,

I asked a similar question earlier [1], but without response. So
indeed, I think there is just a conceptual difference here.

cheers,
Martijn

[1] http://mail.scipy.org/pipermail/ipython-user/2013-February/012233.html





2013/12/6 Tiago Antão <[hidden email]>:

> Dear all,
>
> I have been a long standing user of ipython and ipython notebook. Most of my
> code is run on a grid (SGE mostly, but not only - LSF and Torque sometimes).
>
> I was using my own ad-hoc parallelization code, but decided try to change
> the code to IPython parallel. And I running into what might be some
> conceptual dissonance between the standard grid model and my perception of
> IPython parallel.
>
> So, in a standard grid application (at least my view of it), one submits
> jobs to a queue which will eventually be executed and terminate. The process
> life-cycle starts when the computation starts and ends when the computation
> ends.
>
> It seems to me that IPython parallel (newbie here, I am guessing that I am
> wrong...) imposes that the engines have to be up on each cluster node,
> waiting for computation (even in load balancing mode): They are always on,
> sometimes computing, sometimes just sitting idle waiting for computation. My
> question is: is it possible to have a computing model that is based on
> "standard" grid queues, where an engine is queued for computation, does the
> required computation and dies? For me, asking cluster managers to allow a
> different computing model is not a possible avenue...
>
> I apologize if this is a dead obvious question...
>
> Tiago
>
> --
> "The truth may be out there, but the lies are already in your head" - Terry
> Pratchett
>
> _______________________________________________
> IPython-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel vs "standard" grid computing model

Zoltán Vörös-2
Since ipython-user is sort of being phased out, you might have better
luck by posting the question on ipython-dev.

Cheers,
Zoltán
On 06/12/13 13:28, Martijn Vermaat wrote:

> Hi Tiago,
>
> I asked a similar question earlier [1], but without response. So
> indeed, I think there is just a conceptual difference here.
>
> cheers,
> Martijn
>
> [1] http://mail.scipy.org/pipermail/ipython-user/2013-February/012233.html
>
>

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel vs "standard" grid computing model

Ian Stokes-Rees
In reply to this post by Tiago Antão
On 12/6/13, 7:07 AM, Tiago Antão wrote:
is it possible to have a computing model that is based on "standard" grid queues, where an engine is queued for computation, does the required computation and dies? For me, asking cluster managers to allow a different computing model is not a possible avenue...

IPython Parallel isn't really made for this kind of batch computing, instead it is designed to offer "parallelism" like threading or forking or MPI provide parallelism.  That said, if you know you can get "regular" batch jobs on SGE up and running in <5 minute queue wait time, then you could submit jobs to SGE that then start an ipengine process that will "call-home" to your ipcontroller.  This would (probably) require you to setup *and use* passwordless SSH, in otherwords you'd need to get your ~/.ssh/config set "just so".

I wrote a blog entry about using IPython Parallel back in the summer that provides an IPython profile setup that would probably support this kind of thing.

https://www.wakari.io/sharing/bundle/ijstokes/ipcluster-wakari-intro

If you sign up for a Wakari account you can then see (via the Terminal tab, and then open a "Shell" from the dropdown) in your ~/.ipython/profile_wakari a configuration that gets all the parts of IPython happily configured to work with SSH.  There isn't a lot of documentation on this -- it took me several days to get it right.

Cheers,

Ian

PS - Someone said ipython-user is going away.  What is replacing it?

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel vs "standard" grid computing model

Thomas Kluyver-2
On 6 December 2013 14:03, Ian Stokes-Rees <[hidden email]> wrote:
Someone said ipython-user is going away.  What is replacing it?

We've decided that having separate -user and -dev lists is not actually that helpful - they tend to lead to similar discussions, but some people will only be on one or the other, so they miss out on things. So the ipython-dev list, despite the name, is becoming the combined user+dev list. We're also encouraging questions on Stackoverflow and the Hipchat help room (http://www.hipchat.com/ghSp7E1uY ).

Thomas

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel vs "standard" grid computing model

Paul Hobson
In reply to this post by Ian Stokes-Rees

On Fri, Dec 6, 2013 at 2:03 PM, Ian Stokes-Rees <[hidden email]> wrote:

PS - Someone said ipython-user is going away.  What is replacing it?


ipython-dev 


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user