Quantcast

IPython+zmq and fork()

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

IPython+zmq and fork()

Volker Braun
Sage is getting ready to upgrade to the new IPython, excuses for any email deluge on this list. One question that I have is whether you thought about fork() to quickly spawn children (this is crucial for Sage since starting up a new Sage process is quite slow). According to the zeromq mailinglist, you shouldn't fork after creating a zmq context. This is not just a theoretical problem, I wrote a clustering tool for my own purposes and found out the hard way that bad things can happen if you do.

Will IPython always be usable without zmq?  You are not planning to eventually deprecate the non-zmq console client? Or maybe you have some awesome other solution?



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IPython+zmq and fork()

Jason Grout-5
On 4/12/12 9:43 AM, Volker Braun wrote:

> Sage is getting ready to upgrade to the new IPython, excuses for any email
> deluge on this list. One question that I have is whether you thought about
> fork() to quickly spawn children (this is crucial for Sage since starting up
> a new Sage process is quite slow). According to the zeromq mailinglist, you
> shouldn't fork after creating a zmq context. This is not just a theoretical
> problem, I wrote a clustering tool for my own purposes and found out the
> hard way that bad things can happen if you do.
>
> Will IPython always be usable without zmq?  You are not planning to
> eventually deprecate the non-zmq console client? Or maybe you have some
> awesome other solution?

Just as a point of reference, this documentation page (which seems to be
outdated, maybe?) indicates that a single-user command line session will
eventually use a 2-process model with zmq to communicate between the
processes:

http://ipython.org/ipython-doc/dev/development/ipythonzmq.html

Thanks,

Jason
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IPython+zmq and fork()

MinRK
zmq will never be a dependency for regular IPython.  If we do make basic IPython use the two-process model, it will need to be able to run with stdlib only (there would be no kernel sharing or anything, just frontend surviving segfaults, allowing restart, etc.)

-MinRK

On Apr 12, 2012, at 8:01, Jason Grout <[hidden email]> wrote:

> On 4/12/12 9:43 AM, Volker Braun wrote:
>> Sage is getting ready to upgrade to the new IPython, excuses for any email
>> deluge on this list. One question that I have is whether you thought about
>> fork() to quickly spawn children (this is crucial for Sage since starting up
>> a new Sage process is quite slow). According to the zeromq mailinglist, you
>> shouldn't fork after creating a zmq context. This is not just a theoretical
>> problem, I wrote a clustering tool for my own purposes and found out the
>> hard way that bad things can happen if you do.
>>
>> Will IPython always be usable without zmq?  You are not planning to
>> eventually deprecate the non-zmq console client? Or maybe you have some
>> awesome other solution?
>
> Just as a point of reference, this documentation page (which seems to be
> outdated, maybe?) indicates that a single-user command line session will
> eventually use a 2-process model with zmq to communicate between the
> processes:
>
> http://ipython.org/ipython-doc/dev/development/ipythonzmq.html
>
> Thanks,
>
> Jason
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Embedding IPython+zmq

Ray Osborn
In reply to this post by Jason Grout-5
I didn't want to hijack the other thread so I thought I would create a new one. I just read the page that Jason Grout references about making the two-process iPython standard. I have, I think successfully, adapted Robert Kern's method of creating a kernel that fakes zmq messages within a single process, so that I can embed the iPython shell in a Qt GUI. I need this because I want to display items in the iPython shell namespace in a separate widget and need to trigger Qt signals when certain items in the namespace are changed. As Robert admits, this requires hacking into the zmq code and is probably not as elegant as it could be.

Are there plans to refactor the two-process code to enable a single-process mode easily? Or perhaps zmq messaging could be used to facilitate communication to external GUI widgets, such as providing a way of mapping messages to signals? This is not my area of expertise, so let me know if I have not explained the problem clearly enough.

Thanks,
Ray

Begin forwarded message:

> From: Jason Grout <[hidden email]>
> Subject: Re: [IPython-dev] IPython+zmq and fork()
> Date: April 12, 2012 10:01:25 AM CDT
> To: [hidden email]
> Reply-To: IPython developers list <[hidden email]>
>
> On 4/12/12 9:43 AM, Volker Braun wrote:
>> Sage is getting ready to upgrade to the new IPython, excuses for any email
>> deluge on this list. One question that I have is whether you thought about
>> fork() to quickly spawn children (this is crucial for Sage since starting up
>> a new Sage process is quite slow). According to the zeromq mailinglist, you
>> shouldn't fork after creating a zmq context. This is not just a theoretical
>> problem, I wrote a clustering tool for my own purposes and found out the
>> hard way that bad things can happen if you do.
>>
>> Will IPython always be usable without zmq?  You are not planning to
>> eventually deprecate the non-zmq console client? Or maybe you have some
>> awesome other solution?
>
> Just as a point of reference, this documentation page (which seems to be
> outdated, maybe?) indicates that a single-user command line session will
> eventually use a 2-process model with zmq to communicate between the
> processes:
>
> http://ipython.org/ipython-doc/dev/development/ipythonzmq.html
>
> Thanks,
>
> Jason
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev

--
Ray Osborn
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: [hidden email]



_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IPython+zmq and fork()

Fernando Perez
In reply to this post by Jason Grout-5
On Thu, Apr 12, 2012 at 8:01 AM, Jason Grout
<[hidden email]> wrote:
> Just as a point of reference, this documentation page (which seems to be
> outdated, maybe?) indicates that a single-user command line session will
> eventually use a 2-process model with zmq to communicate between the
> processes:
>
> http://ipython.org/ipython-doc/dev/development/ipythonzmq.html

That already exists, and you can start it with

ipython console

which is capable of connecting to an already running kernel with the
`--existing` flag.

But to clarify, both for the benefit of Sage and any other project
using IPython: we have a policy of keeping the 'classic'
single-process IPython *always* installable with nothing beyond the
standard library (plus pyreadline and setuptools on Windows).  This
hasn't changed now, and there's no sensible reason to change that
ever.  There is enormous value in IPython being able to run just on a
'naked' python installation as a better interactive shell, and many
projects and use cases benefit from this, so we'll never change that
(if we ever accidentally break this, it should be reported as a bug).

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Embedding IPython+zmq

Fernando Perez
In reply to this post by Ray Osborn
Hi Ray,

On Thu, Apr 12, 2012 at 9:44 AM, Ray Osborn <[hidden email]> wrote:
> I didn't want to hijack the other thread so I thought I would create a new one. I just read the page that Jason Grout references about making the two-process iPython standard. I have, I think successfully, adapted Robert Kern's method of creating a kernel that fakes zmq messages within a single process, so that I can embed the iPython shell in a Qt GUI. I need this because I want to display items in the iPython shell namespace in a separate widget and need to trigger Qt signals when certain items in the namespace are changed. As Robert admits, this requires hacking into the zmq code and is probably not as elegant as it could be.
>
> Are there plans to refactor the two-process code to enable a single-process mode easily? Or perhaps zmq messaging could be used to facilitate communication to external GUI widgets, such as providing a way of mapping messages to signals? This is not my area of expertise, so let me know if I have not explained the problem clearly enough.

You have, and indeed I would *love* to have the time for a clean
refactoring of the various kernel/client classes to make it really
trivial to switch from in-process to out-of-process.  In my mind, all
clients should have simply a .kernel attribute that could be either a
LocalKernel or a RemoteKernel, with otherwise identical APIs.  Then,
changing the Qt console from using a remote to a local kernel would be
a trivial one-liner.

We've done bits and pieces of that, but nobody so far has had the
bandwidth to spend on doing this entire refactoring cleanly.
Ultimately that could even produce a notebook that runs without zeromq
(albeit opening only one .ipynb per http port)...

So in summary: yes, we want to do it.  No, it's not on our immediate
work list due to lack of time/higher priorities.

If anyone is interested in doing this right, we'd be happy to review
relevant pull requests, though.

Until then, hacks like what you're using will be the solution, I'm afraid.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Loading...