Quantcast

[IPython-User] ipcluster, profiles, startup scripts and the web notebook

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[IPython-User] ipcluster, profiles, startup scripts and the web notebook

Matthew Turk-3
Hi there,

I'm trying to set up a custom profile for IPython running on my local
machine under MPI, but I'm running into trouble.  This is with the
released 0.13.  I'd like at startup for a couple commands to be run,
to signal to my Python package that it should expect to be running in
parallel (with MPI).  Unfortunately, I'm not quite getting the
behavior I'd expect, and I think it must be something simple on my end
that I'm not understanding.

I've created a new profile with:

ipython profile create --profile yt --parallel

Into the startup directory I've placed the normal startup commands --
in this case, the main import for the module I intend to work with --
inside a 00-startup.py file.  I'm able to spawn new notebooks that
start up as I'd expect if I do "ipython notebook --profile=yt".  So
that seems to work properly.  For starting in parallel, I've modified
the generated profile in a few places:

 * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
 * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
ipcluster_config.py
 * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
not sure if this was either unnecessary or redundant)
 * In ipengine_config.py I've added a startup command in
c.IPEngineApp.startup_command

In the web notebook launcher, changes to the number of processors is
reflected in what happens when I click on "start".  And looking at the
output of top/ps I also see that the engines have been launched with
mpiexec as I have requested.  I'm also able to connect to them and
verify that they have been launched, but running "dir()" doesn't show
the expected imports, from the startup_command.  I'm also unable to
get anything to be run if I set c.MPI.init_script to be an existing
script.  Am I missing something with how to set up the configuration?

Thanks for any ideas,

Matt
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipcluster, profiles, startup scripts and the web notebook

MinRK


On Fri, Jul 27, 2012 at 10:53 AM, Matthew Turk <[hidden email]> wrote:
Hi there,

I'm trying to set up a custom profile for IPython running on my local
machine under MPI, but I'm running into trouble.  This is with the
released 0.13.  I'd like at startup for a couple commands to be run,
to signal to my Python package that it should expect to be running in
parallel (with MPI).  Unfortunately, I'm not quite getting the
behavior I'd expect, and I think it must be something simple on my end
that I'm not understanding.

I've created a new profile with:

ipython profile create --profile yt --parallel

Into the startup directory I've placed the normal startup commands --
in this case, the main import for the module I intend to work with --
inside a 00-startup.py file.  I'm able to spawn new notebooks that
start up as I'd expect if I do "ipython notebook --profile=yt".  So
that seems to work properly.  For starting in parallel, I've modified
the generated profile in a few places:

 * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
 * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
ipcluster_config.py

Exactly right.
 
 * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
not sure if this was either unnecessary or redundant)

Yes, setting profile in config files is indeed always redundant (both cases, here).
 
 * In ipengine_config.py I've added a startup command in
c.IPEngineApp.startup_command

Re-reading the code, this config was broken when the forked Kernel implementations were merged for 0.13.  
I think right now it is impossible to trigger startup code on engines in 0.13 or master.
Can you open an Issue for this?

The reason being that Kernels/Shells are shared, but the Application objects are still separate, and that is where this startup config lives.  Ultimately, IPEngineApp should go away, but we aren't quite there yet.


In the web notebook launcher, changes to the number of processors is
reflected in what happens when I click on "start".  And looking at the
output of top/ps I also see that the engines have been launched with
mpiexec as I have requested.  I'm also able to connect to them and
verify that they have been launched, but running "dir()" doesn't show
the expected imports, from the startup_command.  I'm also unable to
get anything to be run if I set c.MPI.init_script to be an existing
script.  Am I missing something with how to set up the configuration?

Thanks for any ideas,

Matt
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipcluster, profiles, startup scripts and the web notebook

Matthew Turk-3
Hi Min,

On Fri, Jul 27, 2012 at 4:04 PM, MinRK <[hidden email]> wrote:

>
>
> On Fri, Jul 27, 2012 at 10:53 AM, Matthew Turk <[hidden email]>
> wrote:
>>
>> Hi there,
>>
>> I'm trying to set up a custom profile for IPython running on my local
>> machine under MPI, but I'm running into trouble.  This is with the
>> released 0.13.  I'd like at startup for a couple commands to be run,
>> to signal to my Python package that it should expect to be running in
>> parallel (with MPI).  Unfortunately, I'm not quite getting the
>> behavior I'd expect, and I think it must be something simple on my end
>> that I'm not understanding.
>>
>> I've created a new profile with:
>>
>> ipython profile create --profile yt --parallel
>>
>> Into the startup directory I've placed the normal startup commands --
>> in this case, the main import for the module I intend to work with --
>> inside a 00-startup.py file.  I'm able to spawn new notebooks that
>> start up as I'd expect if I do "ipython notebook --profile=yt".  So
>> that seems to work properly.  For starting in parallel, I've modified
>> the generated profile in a few places:
>>
>>  * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
>>  * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
>> ipcluster_config.py
>
>
> Exactly right.
>
>>
>>  * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
>> not sure if this was either unnecessary or redundant)
>
>
> Yes, setting profile in config files is indeed always redundant (both cases,
> here).
>
>>
>>  * In ipengine_config.py I've added a startup command in
>> c.IPEngineApp.startup_command
>
>
> Re-reading the code, this config was broken when the forked Kernel
> implementations were merged for 0.13.
> I think right now it is impossible to trigger startup code on engines in
> 0.13 or master.
> Can you open an Issue for this?
>
> The reason being that Kernels/Shells are shared, but the Application objects
> are still separate, and that is where this startup config lives.
> Ultimately, IPEngineApp should go away, but we aren't quite there yet.

Ah, okay.  Thanks for digging into this.  I've opened issue #2213:

https://github.com/ipython/ipython/issues/2213

Best,

Matt

>
>>
>> In the web notebook launcher, changes to the number of processors is
>> reflected in what happens when I click on "start".  And looking at the
>> output of top/ps I also see that the engines have been launched with
>> mpiexec as I have requested.  I'm also able to connect to them and
>> verify that they have been launched, but running "dir()" doesn't show
>> the expected imports, from the startup_command.  I'm also unable to
>> get anything to be run if I set c.MPI.init_script to be an existing
>> script.  Am I missing something with how to set up the configuration?
>>
>> Thanks for any ideas,
>>
>> Matt
>> _______________________________________________
>> IPython-User mailing list
>> [hidden email]
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipcluster, profiles, startup scripts and the web notebook

MinRK


On Fri, Jul 27, 2012 at 2:19 PM, Matthew Turk <[hidden email]> wrote:
Hi Min,

On Fri, Jul 27, 2012 at 4:04 PM, MinRK <[hidden email]> wrote:
>
>
> On Fri, Jul 27, 2012 at 10:53 AM, Matthew Turk <[hidden email]>
> wrote:
>>
>> Hi there,
>>
>> I'm trying to set up a custom profile for IPython running on my local
>> machine under MPI, but I'm running into trouble.  This is with the
>> released 0.13.  I'd like at startup for a couple commands to be run,
>> to signal to my Python package that it should expect to be running in
>> parallel (with MPI).  Unfortunately, I'm not quite getting the
>> behavior I'd expect, and I think it must be something simple on my end
>> that I'm not understanding.
>>
>> I've created a new profile with:
>>
>> ipython profile create --profile yt --parallel
>>
>> Into the startup directory I've placed the normal startup commands --
>> in this case, the main import for the module I intend to work with --
>> inside a 00-startup.py file.  I'm able to spawn new notebooks that
>> start up as I'd expect if I do "ipython notebook --profile=yt".  So
>> that seems to work properly.  For starting in parallel, I've modified
>> the generated profile in a few places:
>>
>>  * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
>>  * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
>> ipcluster_config.py
>
>
> Exactly right.
>
>>
>>  * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
>> not sure if this was either unnecessary or redundant)
>
>
> Yes, setting profile in config files is indeed always redundant (both cases,
> here).
>
>>
>>  * In ipengine_config.py I've added a startup command in
>> c.IPEngineApp.startup_command
>
>
> Re-reading the code, this config was broken when the forked Kernel
> implementations were merged for 0.13.
> I think right now it is impossible to trigger startup code on engines in
> 0.13 or master.
> Can you open an Issue for this?
>
> The reason being that Kernels/Shells are shared, but the Application objects
> are still separate, and that is where this startup config lives.
> Ultimately, IPEngineApp should go away, but we aren't quite there yet.

Ah, okay.  Thanks for digging into this.  I've opened issue #2213:

https://github.com/ipython/ipython/issues/2213

Thanks!  And I may have a fix for you.
 


Best,

Matt

>
>>
>> In the web notebook launcher, changes to the number of processors is
>> reflected in what happens when I click on "start".  And looking at the
>> output of top/ps I also see that the engines have been launched with
>> mpiexec as I have requested.  I'm also able to connect to them and
>> verify that they have been launched, but running "dir()" doesn't show
>> the expected imports, from the startup_command.  I'm also unable to
>> get anything to be run if I set c.MPI.init_script to be an existing
>> script.  Am I missing something with how to set up the configuration?
>>
>> Thanks for any ideas,
>>
>> Matt
>> _______________________________________________
>> IPython-User mailing list
>> [hidden email]
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user


_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipcluster, profiles, startup scripts and the web notebook

Matthew Turk-3
Hi Min

On Fri, Jul 27, 2012 at 5:31 PM, MinRK <[hidden email]> wrote:

>
>
> On Fri, Jul 27, 2012 at 2:19 PM, Matthew Turk <[hidden email]> wrote:
>>
>> Hi Min,
>>
>> On Fri, Jul 27, 2012 at 4:04 PM, MinRK <[hidden email]> wrote:
>> >
>> >
>> > On Fri, Jul 27, 2012 at 10:53 AM, Matthew Turk <[hidden email]>
>> > wrote:
>> >>
>> >> Hi there,
>> >>
>> >> I'm trying to set up a custom profile for IPython running on my local
>> >> machine under MPI, but I'm running into trouble.  This is with the
>> >> released 0.13.  I'd like at startup for a couple commands to be run,
>> >> to signal to my Python package that it should expect to be running in
>> >> parallel (with MPI).  Unfortunately, I'm not quite getting the
>> >> behavior I'd expect, and I think it must be something simple on my end
>> >> that I'm not understanding.
>> >>
>> >> I've created a new profile with:
>> >>
>> >> ipython profile create --profile yt --parallel
>> >>
>> >> Into the startup directory I've placed the normal startup commands --
>> >> in this case, the main import for the module I intend to work with --
>> >> inside a 00-startup.py file.  I'm able to spawn new notebooks that
>> >> start up as I'd expect if I do "ipython notebook --profile=yt".  So
>> >> that seems to work properly.  For starting in parallel, I've modified
>> >> the generated profile in a few places:
>> >>
>> >>  * I've set c.IPClusterEngines.profile = u'yt' in ipcluster_config.py
>> >>  * I've changed c.IPClusterEngines.engine_launcher_class to "MPI" in
>> >> ipcluster_config.py
>> >
>> >
>> > Exactly right.
>> >
>> >>
>> >>  * In ipengine_config.py I've set c.IPEngineApp.profile = u'yt' (I was
>> >> not sure if this was either unnecessary or redundant)
>> >
>> >
>> > Yes, setting profile in config files is indeed always redundant (both
>> > cases,
>> > here).
>> >
>> >>
>> >>  * In ipengine_config.py I've added a startup command in
>> >> c.IPEngineApp.startup_command
>> >
>> >
>> > Re-reading the code, this config was broken when the forked Kernel
>> > implementations were merged for 0.13.
>> > I think right now it is impossible to trigger startup code on engines in
>> > 0.13 or master.
>> > Can you open an Issue for this?
>> >
>> > The reason being that Kernels/Shells are shared, but the Application
>> > objects
>> > are still separate, and that is where this startup config lives.
>> > Ultimately, IPEngineApp should go away, but we aren't quite there yet.
>>
>> Ah, okay.  Thanks for digging into this.  I've opened issue #2213:
>>
>> https://github.com/ipython/ipython/issues/2213
>
>
> Thanks!  And I may have a fix for you.
>

Those changes fixed it for me when applied to 0.13.  Thanks!

-Matt

>>
>>
>>
>> Best,
>>
>> Matt
>>
>> >
>> >>
>> >> In the web notebook launcher, changes to the number of processors is
>> >> reflected in what happens when I click on "start".  And looking at the
>> >> output of top/ps I also see that the engines have been launched with
>> >> mpiexec as I have requested.  I'm also able to connect to them and
>> >> verify that they have been launched, but running "dir()" doesn't show
>> >> the expected imports, from the startup_command.  I'm also unable to
>> >> get anything to be run if I set c.MPI.init_script to be an existing
>> >> script.  Am I missing something with how to set up the configuration?
>> >>
>> >> Thanks for any ideas,
>> >>
>> >> Matt
>> >> _______________________________________________
>> >> IPython-User mailing list
>> >> [hidden email]
>> >> http://mail.scipy.org/mailman/listinfo/ipython-user
>> >
>> >
>> >
>> > _______________________________________________
>> > IPython-User mailing list
>> > [hidden email]
>> > http://mail.scipy.org/mailman/listinfo/ipython-user
>> >
>> _______________________________________________
>> IPython-User mailing list
>> [hidden email]
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
>
> _______________________________________________
> IPython-User mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipcluster, profiles, startup scripts and the web notebook

alfred
In reply to this post by MinRK
Is this issue really fixed? I'm using the fix, and I've been trying to execute a startup script in the engines (running on my local machine), but it is not working. My ipengine_config.py is as simple as this:

c = get_config()
c.IPEngineApp.startup_script = '/path/to/startup.py'

The code I have in the script does not seem to be executed. For example, I try to do a map_sync using a function that uses some modules imported in the script - without importing them again - but it doesn't work. They don't even work when I use startup_command.

Am I doing something wrong?
Anyone else having issues with that?
Loading...