ipython1 saw exception transport and raising

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

ipython1 saw exception transport and raising

dfj225@gmail.com
Hello IPython devs,

I've been testing the ipython1 saw branch and was testing some of the
new exception handling code. All in all, it seems to be working very
well, but I have come across one quirk that affects my code. Part of
the system that I use involves C++ code wrapped using Boost.Python.
For some reason, it seems that Boost.Python likes to create its own
classes for exceptions which aren't able to be pickled. So, whenever
one of these exceptions gets generated, I get a traceback about failed
pickling instead of the actual error that was generated.

I cooked up a little change that makes these types of errors able to
be serialized. I'm not too familiar with the internals of IPython1, so
there might be a much better way to do this, but at the very least I
hope my code serves as a good indication of where the problem lies.

Thanks,
~doug

_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev

pbutil_exception.diff (980 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Brian Granger-2
Doug,

Can you do an svn up and try again.  You patch made it pretty clear
what the problem was.  However, I did make some changes to make sure
that the new type and value are something other than a string.  Here
is what I committed:

def packageFailure(f):
    """Clean and pickle a failure preappending the string FAILURE:"""

    f.cleanFailure()
    # This is sometimes helpful in debugging
    #f.raiseException()
    try:
        pString = pickle.dumps(f, 2)
    except pickle.PicklingError:
        # Certain types of exceptions are not pickleable, for instance ones
        # from Boost.Python.  We try to wrap them in something that is
        f.type = UnpickleableException
        f.value = UnpickleableException(str(f.type) + ": " + str(f.value))
        pString = pickle.dumps(f, 2)
    return 'FAILURE:' + pString

Let me know how this works - I don't have Boost.Python installed so I
can't reproduce your situation exactly.  I can modify it further if
needed.

Brian

On 7/9/07, Doug Jones <[hidden email]> wrote:

> Hello IPython devs,
>
> I've been testing the ipython1 saw branch and was testing some of the
> new exception handling code. All in all, it seems to be working very
> well, but I have come across one quirk that affects my code. Part of
> the system that I use involves C++ code wrapped using Boost.Python.
> For some reason, it seems that Boost.Python likes to create its own
> classes for exceptions which aren't able to be pickled. So, whenever
> one of these exceptions gets generated, I get a traceback about failed
> pickling instead of the actual error that was generated.
>
> I cooked up a little change that makes these types of errors able to
> be serialized. I'm not too familiar with the internals of IPython1, so
> there might be a much better way to do this, but at the very least I
> hope my code serves as a good indication of where the problem lies.
>
> Thanks,
> ~doug
>
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>
>
>
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

dfj225@gmail.com
Hi Brian,

Thanks for looking into this. I did the svn up and tested it and
everything looks fine now. The  result is more or less the same as
what I cooked up, meaning the name of the original exception is
displayed in the traceback.

However, there still seems to be some data loss in the process. Let me
demonstrate what I mean.

If I use my module in IPython itself and do something that I know
invokes an exception from some code that uses Boost.Python, I get
output like so:

In [3]: a = attribute_random()
---------------------------------------------------------------------------
Boost.Python.ArgumentError                             Traceback (most
recent call last)

<cut traceback>

ArgumentError: Python argument types in
    _basin.attribute_random()
did not match C++ signature:
    attribute_random(int b, unsigned int seed)
    attribute_random(std::vector<int, std::allocator<int> > b,
unsigned int seed)

I've included this to give you some insight to how this particular
Exception is formed:

In [2]: try:
   ...:     a = attribute_random()
   ...: except Exception, inst:
   ...:     print type(inst)
   ...:     print inst.args
   ...:     print inst
   ...:
<type 'instance'>
('Python argument types in\n    _basin.attribute_random()\ndid not
match C++ signature:\n    attribute_random(int b, unsigned int seed)\n
   attribute_random(std::vector<int, std::allocator<int> > b, unsigned
int seed)',)
Python argument types in
    _basin.attribute_random()
did not match C++ signature:
    attribute_random(int b, unsigned int seed)
    attribute_random(std::vector<int, std::allocator<int> > b,
unsigned int seed)

Now, when this same Exception is generated in a remote session, I get
output like so:

In [6]: a = attribute_random()
2007/07/16 13:56 -0400 [-] Performing execute on all
---------------------------------------------------------------------------
ipython1.kernel.error.UnpickleableException
Traceback (most recent call last)

<cut traceback>

UnpickleableException: ipython1.kernel.error.UnpickleableException:
***************************************************************************
An exception occurred in the IPython Interpreter due to a user action.
This usually means that user code has a problem in it somewhere.

engine: 1
method: execute(lines)
lines = a = attribute_random()

A full traceback from the actual interpreter:
---------------------------------------------------------------------------
Boost.Python.ArgumentError                             Traceback (most
recent call last)


ArgumentError:
***************************************************************************

As you see, some of the information that is part of the ArgumentError
is either not transported or not displayed. Unlike last time, I don't
really have any ideas as to why.


Thanks,
~Doug

On 7/12/07, Brian Granger <[hidden email]> wrote:

> Doug,
>
> Can you do an svn up and try again.  You patch made it pretty clear
> what the problem was.  However, I did make some changes to make sure
> that the new type and value are something other than a string.  Here
> is what I committed:
>
> def packageFailure(f):
>     """Clean and pickle a failure preappending the string FAILURE:"""
>
>     f.cleanFailure()
>     # This is sometimes helpful in debugging
>     #f.raiseException()
>     try:
>         pString = pickle.dumps(f, 2)
>     except pickle.PicklingError:
>         # Certain types of exceptions are not pickleable, for instance ones
>         # from Boost.Python.  We try to wrap them in something that is
>         f.type = UnpickleableException
>         f.value = UnpickleableException(str(f.type) + ": " + str(f.value))
>         pString = pickle.dumps(f, 2)
>     return 'FAILURE:' + pString
>
> Let me know how this works - I don't have Boost.Python installed so I
> can't reproduce your situation exactly.  I can modify it further if
> needed.
>
> Brian
>
> On 7/9/07, Doug Jones <[hidden email]> wrote:
> > Hello IPython devs,
> >
> > I've been testing the ipython1 saw branch and was testing some of the
> > new exception handling code. All in all, it seems to be working very
> > well, but I have come across one quirk that affects my code. Part of
> > the system that I use involves C++ code wrapped using Boost.Python.
> > For some reason, it seems that Boost.Python likes to create its own
> > classes for exceptions which aren't able to be pickled. So, whenever
> > one of these exceptions gets generated, I get a traceback about failed
> > pickling instead of the actual error that was generated.
> >
> > I cooked up a little change that makes these types of errors able to
> > be serialized. I'm not too familiar with the internals of IPython1, so
> > there might be a much better way to do this, but at the very least I
> > hope my code serves as a good indication of where the problem lies.
> >
> > Thanks,
> > ~doug
> >
> > _______________________________________________
> > IPython-dev mailing list
> > [hidden email]
> > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
> >
> >
> >
>
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
On 7/16/07, Doug Jones <[hidden email]> wrote:
> Hi Brian,
>
> Thanks for looking into this. I did the svn up and tested it and
> everything looks fine now. The  result is more or less the same as
> what I cooked up, meaning the name of the original exception is
> displayed in the traceback.
>
> However, there still seems to be some data loss in the process. Let me
> demonstrate what I mean.

Yup, this is in my immediate to-do list, to which I think I'll be able
to get to on Wednesday, as I'll finally get a small break from various
work-related things.

If Brian doesn't beat me to it, I hope to get this fixed soon, as it's
a *major* problem we have right now in terms of using the system.

If you have a small self-contained example that shows the problem and
does not require Boost, please send it this way, it will help to add
it to the test suite.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Brian Granger-2
I am headed on vacation soon, so if Fernando can take this one that
would be great.

Thanks

Brian

On 7/16/07, Fernando Perez <[hidden email]> wrote:

> On 7/16/07, Doug Jones <[hidden email]> wrote:
> > Hi Brian,
> >
> > Thanks for looking into this. I did the svn up and tested it and
> > everything looks fine now. The  result is more or less the same as
> > what I cooked up, meaning the name of the original exception is
> > displayed in the traceback.
> >
> > However, there still seems to be some data loss in the process. Let me
> > demonstrate what I mean.
>
> Yup, this is in my immediate to-do list, to which I think I'll be able
> to get to on Wednesday, as I'll finally get a small break from various
> work-related things.
>
> If Brian doesn't beat me to it, I hope to get this fixed soon, as it's
> a *major* problem we have right now in terms of using the system.
>
> If you have a small self-contained example that shows the problem and
> does not require Boost, please send it this way, it will help to add
> it to the test suite.
>
> Cheers,
>
> f
>
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

dfj225@gmail.com
In reply to this post by Fernando Perez
For a test, would something like this be sufficient:

raise Exception("Something extra")

in a normal ipython shell I get this:

In [16]: raise Exception("Something extra")
---------------------------------------------------------------------------
exceptions.Exception                                 Traceback (most
recent call last)

/home/djones/new_svn/basin_remote/trunk/usr/lib64/python2.4/site-packages/<ipython
console>

Exception: Something extra

Using rc.executeAll(), I get:

In [17]: rc.executeAll('raise Exception("Some extra info.")')
2007/07/16 18:57 -0400 [-] Performing execute on all
---------------------------------------------------------------------------
exceptions.Exception                                 Traceback (most
recent call last)

/home/djones/new_svn/basin_remote/trunk/usr/lib64/python2.4/site-packages/<ipython
console>

<cut traceback>

Exception:
***************************************************************************
An exception occurred in the IPython Interpreter due to a user action.
This usually means that user code has a problem in it somewhere.

engine: 0
method: execute(lines)
lines = raise Exception("Some extra info.")

A full traceback from the actual interpreter:
---------------------------------------------------------------------------
exceptions.Exception                                 Traceback (most
recent call last)


Exception:
***************************************************************************

Note the missing "Some extra info".


~doug



On 7/16/07, Fernando Perez <[hidden email]> wrote:

> On 7/16/07, Doug Jones <[hidden email]> wrote:
> > Hi Brian,
> >
> > Thanks for looking into this. I did the svn up and tested it and
> > everything looks fine now. The  result is more or less the same as
> > what I cooked up, meaning the name of the original exception is
> > displayed in the traceback.
> >
> > However, there still seems to be some data loss in the process. Let me
> > demonstrate what I mean.
>
> Yup, this is in my immediate to-do list, to which I think I'll be able
> to get to on Wednesday, as I'll finally get a small break from various
> work-related things.
>
> If Brian doesn't beat me to it, I hope to get this fixed soon, as it's
> a *major* problem we have right now in terms of using the system.
>
> If you have a small self-contained example that shows the problem and
> does not require Boost, please send it this way, it will help to add
> it to the test suite.
>
> Cheers,
>
> f
>
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
On 7/16/07, Doug Jones <[hidden email]> wrote:
> For a test, would something like this be sufficient:
>
> raise Exception("Something extra")
[...]
> Note the missing "Some extra info".


That will do it, thanks.  My local copy of saw is currently totally
borked due to changes I'm in the middle of, so I just wanted to make
sure we had a little easy-to-see example.  I'll take care of it later
this week and will post here.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
Hi Doug,

On 8/10/07, Doug Jones <[hidden email]> wrote:
> I didn't see any messages in response to this, so I'm curious, has
> this been fixed in svn yet?

I'm afraid no, and I'm embarrassed to say so.  All my recent time for
ipython went into two large pieces of infrastructure that were
long-needed and that required a solid, concentrated effort (the
TConfig system and the SnakeOil testing tools, both now in the
sandbox).

That was roughly 6000 LOC I had to write in just the last 3 weeks, and
 it required a lot of concentrated effort.

Those are now both finished, and I  hope that this week while at the
Scipy conference, Brian and I will be able to pound a bit on these
remaining problems.

My apology: I know the exception propagation bug is *critical* and I'd
promised to fix it long ago.  But there's only so much code I can
write...

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
On 8/11/07, Fernando Perez <[hidden email]> wrote:

> That was roughly 6000 LOC I had to write in just the last 3 weeks, and
                   ^^^^
Oops, typo: just 4000.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
In reply to this post by dfj225@gmail.com
On 7/16/07, Doug Jones <[hidden email]> wrote:

> As you see, some of the information that is part of the ArgumentError
> is either not transported or not displayed. Unlike last time, I don't
> really have any ideas as to why.

OK, I think I got it finally... Please svn up and let us know.  Here's
what I'm getting for simple tests:

ZeroDivisionError: integer division or modulo by zero
***************************************************************************
Exception in the IPython Engine.

Engine action that caused the error:

engine: 0
method: execute(lines)
lines =
import os
os.chdir('/home/fperez/ipython/test/ip1')
import err
err.bad()
#abd snytxx


Full traceback:
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)

/home/fperez/ipython/test/ip1/<string> in <module>()

/home/fperez/ipython/test/ip1/err.py in bad()
     14
     15 def bad():
---> 16     1/0
     17
     18 #bad()

ZeroDivisionError: integer division or modulo by zero
***************************************************************************

WARNING: Failure executing file: <tst.py>


Syntax errors are reported with a clumsy traceback:

SyntaxError: invalid syntax (line 5)
***************************************************************************
Exception in the IPython Engine.

Engine action that caused the error:

engine: 0
method: execute(lines)
lines =
import os
os.chdir('/home/fperez/ipython/test/ip1')
import err
err.bad()
abd snytxx


Full traceback:
---------------------------------------------------------------------------
SyntaxError                               Traceback (most recent call last)

/home/fperez/usr/lib/python2.5/site-packages/ipython1/core/interpreter.py
in split_commands(self, python)
    511         # Uncomment to help debug the ast tree
    512         # for n in ast.node:
--> 513         #     print n.lineno,'->',n
    514
    515         # Each separate command is available by iterating over
ast.node. The

/home/fperez/ipython/test/ip1/compiler/transformer.py in parse(buf, mode)
     50
     51
---> 52
     53
     54

/home/fperez/ipython/test/ip1/compiler/transformer.py in parsesuite(self, text)
    127
    128
--> 129
    130
    131

SyntaxError: invalid syntax (line 5)
***************************************************************************


WARNING: Failure executing file: <tst.py>


But the info is really  all there.   It's just that managing traceback
formatting across the network in this way is not particularly easy,
and I'm a bit too fried now to do more...

But at least now we are getting all the necessary info across, even if
for syntax errors the report isn't the nicest.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

dfj225@gmail.com
Fernando,

First, sorry that it has taken me so long to respond.

Second, I have updated my ipython1 source and the behavior on my end
seems to be correct as well. I am testing with exceptions generated
from Boost.Python and now all information associated with the
exceptions seems to be displayed, including the correct traceback on
the engine.

Thank you,
~Doug

On 8/15/07, Fernando Perez <[hidden email]> wrote:

> On 7/16/07, Doug Jones <[hidden email]> wrote:
>
> > As you see, some of the information that is part of the ArgumentError
> > is either not transported or not displayed. Unlike last time, I don't
> > really have any ideas as to why.
>
> OK, I think I got it finally... Please svn up and let us know.  Here's
> what I'm getting for simple tests:
>
> ZeroDivisionError: integer division or modulo by zero
> ***************************************************************************
> Exception in the IPython Engine.
>
> Engine action that caused the error:
>
> engine: 0
> method: execute(lines)
> lines =
> import os
> os.chdir('/home/fperez/ipython/test/ip1')
> import err
> err.bad()
> #abd snytxx
>
>
> Full traceback:
> ---------------------------------------------------------------------------
> ZeroDivisionError                         Traceback (most recent call last)
>
> /home/fperez/ipython/test/ip1/<string> in <module>()
>
> /home/fperez/ipython/test/ip1/err.py in bad()
>      14
>      15 def bad():
> ---> 16     1/0
>      17
>      18 #bad()
>
> ZeroDivisionError: integer division or modulo by zero
> ***************************************************************************
>
> WARNING: Failure executing file: <tst.py>
>
>
> Syntax errors are reported with a clumsy traceback:
>
> SyntaxError: invalid syntax (line 5)
> ***************************************************************************
> Exception in the IPython Engine.
>
> Engine action that caused the error:
>
> engine: 0
> method: execute(lines)
> lines =
> import os
> os.chdir('/home/fperez/ipython/test/ip1')
> import err
> err.bad()
> abd snytxx
>
>
> Full traceback:
> ---------------------------------------------------------------------------
> SyntaxError                               Traceback (most recent call last)
>
> /home/fperez/usr/lib/python2.5/site-packages/ipython1/core/interpreter.py
> in split_commands(self, python)
>     511         # Uncomment to help debug the ast tree
>     512         # for n in ast.node:
> --> 513         #     print n.lineno,'->',n
>     514
>     515         # Each separate command is available by iterating over
> ast.node. The
>
> /home/fperez/ipython/test/ip1/compiler/transformer.py in parse(buf, mode)
>      50
>      51
> ---> 52
>      53
>      54
>
> /home/fperez/ipython/test/ip1/compiler/transformer.py in parsesuite(self, text)
>     127
>     128
> --> 129
>     130
>     131
>
> SyntaxError: invalid syntax (line 5)
> ***************************************************************************
>
>
> WARNING: Failure executing file: <tst.py>
>
>
> But the info is really  all there.   It's just that managing traceback
> formatting across the network in this way is not particularly easy,
> and I'm a bit too fried now to do more...
>
> But at least now we are getting all the necessary info across, even if
> for syntax errors the report isn't the nicest.
>
> Cheers,
>
> f
>
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: ipython1 saw exception transport and raising

Fernando Perez
On 8/27/07, Doug Jones <[hidden email]> wrote:
> Fernando,
>
> First, sorry that it has taken me so long to respond.
>
> Second, I have updated my ipython1 source and the behavior on my end
> seems to be correct as well. I am testing with exceptions generated
> from Boost.Python and now all information associated with the
> exceptions seems to be displayed, including the correct traceback on
> the engine.

Great, glad we got this one.  Let us know of any other issues you may see.

Cheers,

f
_______________________________________________
IPython-dev mailing list
[hidden email]
http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev