Unintuitive comparison between long and BigInteger

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Unintuitive comparison between long and BigInteger

Johan Van Noten

Jeff,

 

Verifying your fix was blocked by the issue handled in thread “[Jython-users] 2.7b3 JSR-223 ScriptEngine == null (b2 works)”.

That one is now solved, so I retested your fix.

 

Result = success J

 

Thanks for the solution,

Johan

 

Van: Johan Van Noten [mailto:[hidden email]]
Verzonden: woensdag 15 oktober 2014 14:08
Aan: Jeff Allen; [hidden email]
Onderwerp: Re: [Jython-users] Unintuitive comparison between long and BigInteger

 

Hi Jeff,

 

Glad I could help. Thanks for the quick solution!

 

I will verify the solution in the real target app once I get through another Jython 2.7 b3 issue (see separate post).

 

Regards,

Johan

 

Van: Jeff Allen [[hidden email]]
Verzonden: woensdag 15 oktober 2014 0:19
Aan:
[hidden email]
Onderwerp: Re: [Jython-users] Unintuitive comparison between long and BigInteger

 

Johan:

I think this is now fixed under http://bugs.jython.org/issue2090, using the code contributed there. Thanks for helping us to understand it.

Santoso's observation is closely related, but not quite the same thing, because of differences in the way the JSR-223 and regular interpreters store and retrieve variables.

Jeff Allen

On 09/10/2014 23:44, Jeff Allen wrote:

This is very odd. Using essentially Johan's code:

TEST >>> a = 13L
None
TEST >>> repr(type(a))
<type 'java.math.BigInteger'>
TEST >>> repr(type(10L))
<type 'long'>

This is not to be confused with:
TEST >>> type(10L)
class org.python.core.PyLong

That happens, rather than <type 'long'>, because the output implicitly calls toString() instead or repr(), which a real console would.

It seems that values assigned to variables are stripped of their Jython wrapping. I believe this happens here:
https://hg.python.org/jython/file/1e8dd809df28/src/org/python/jsr223/PyScriptEngineScope.java#l147

The call of __tojava__ looks deliberate, but why? Sorry, it's late in the UK.

Jeff Allen

On 09/10/2014 21:23, Curtis Rueden wrote:

Hi Johan,

 

> This behavior doesn’t seem to occur in the default Jython console.

> Same instructions, correct result. As my simple example shows, this

> behavior does happen on Jython script evaluations through Java’s

> ScriptEngine interface.

 

Yep, it seems specific to org.python.jsr223.PyScriptEngine.

 

The reason the Fiji project I mentioned does not have the issue, even though it uses Jython in a JSR-223 context as well, is because that project uses the SciJava scripting framework's implementation [1] which wraps Jython's JSR-223 code _except_ for the ScriptEngine (probably because it predates Jython's official JSR-223 code). Specifically, the SciJava script engine evaluates Jython code as follows:

 

 

I tried tracing through the execution of your minimal script in the debugger in Eclipse, but I couldn't ultimately see what was going on, since the code gets compiled to Java bytecode before execution.

 

So I'm not sure what causes the difference, but if you need JSR-223 support and want to avoid this bug, a simple workaround would be to use the SciJava scripting framework's engine instead.

 

Regards,

Curtis


<big snip>

On Thu, Oct 9, 2014 at 1:19 PM, Santoso Wijaya <[hidden email]> wrote:

More succinctly, this:

 

Jython 2.7b3+ (default:2c45f75a5406, Oct 7 2014, 17:21:51) 

[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_67

Type "help", "copyright", "credits" or "license" for more information.

>>> from java.math import BigInteger

>>> a = BigInteger('10')

>>> type(a)

<type 'java.math.BigInteger'>

>>> a > 100

True

 


~/santoso

 

On Thu, Oct 9, 2014 at 2:24 AM, Johan Van Noten <[hidden email]> wrote:

Hi Jeff,

This issue keeps bugging me... also in 2.7-b3.
So I decided to make a small Java application to demonstrate the issue.

Some easy steps:
1) Download archive from http://www.qtc.be/TestJython.zip
2) Extract
3) Run the script TestJython.cmd (just runs a small class with the jython library on the cp)
4) Enter the example data (see below)
5) It is a small Eclipse Kepler project, so if you prefer, you can load/debug/run it from Eclipse.

When I do this, I get:
E:\tmp\Jython\TestJython>TestJython.cmd
TestJython >>> 10L > 100
false
TestJython >>> a = 10L
None
TestJython >>> a > 100
true
TestJython >>> type(a)
class java.math.BigInteger
TestJython >>> exit

So, as I confirmed initially, the underlying layers seem to use the BigInteger class, causing the failure in comparison.
As far as I know, I am not doing anything specific that could cause this.
I just use Jython as a Python Scripting Engine according to JSR-223, which is a convenient way of using it.
(Take a look at the extremely simple src/TestJython.java)

Do you expect this behavior?
Can it be avoided?

Johan

 

 


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: Unintuitive comparison between long and BigInteger

Curtis Rueden
Hi everyone,

Jeff Allen wrote:
> We now have, using Raphael's patch and an embedded JSR-233 script engine:
> TEST >>> from java.math import BigInteger
> TEST >>> a = BigInteger('10')
> TEST >>> type(a)
> class org.python.core.PyLong

Apologies if I'm being naive, but wouldn't the ultimately desired behavior be as follows?

TEST >>> from java.math import BigInteger
TEST >>> a = BigInteger('10')
TEST >>> type(a)
class java.math.BigInteger
TEST >>> a = 10L
TEST >>> type(a)
class org.python.core.PyLong

Otherwise, intuitively there might be problems in the "other direction" with legitimate BigIntegers being misinterpreted somehow, right?

The SciJava Jython script engine I mentioned earlier has the following behavior:

from java.math import BigInteger
a = BigInteger('10')
print(type(a))
a = 10L
print(type(a))

Produces:

<type 'java.math.BigInteger'>
<type 'long'>

Regards,
Curtis

On Wed, Oct 15, 2014 at 3:43 PM, Johan Van Noten <[hidden email]> wrote:

Jeff,

 

Verifying your fix was blocked by the issue handled in thread “[Jython-users] 2.7b3 JSR-223 ScriptEngine == null (b2 works)”.

That one is now solved, so I retested your fix.

 

Result = success J

 

Thanks for the solution,

Johan

 

Van: Johan Van Noten [mailto:[hidden email]]
Verzonden: woensdag 15 oktober 2014 14:08
Aan: Jeff Allen; [hidden email]


Onderwerp: Re: [Jython-users] Unintuitive comparison between long and BigInteger

 

Hi Jeff,

 

Glad I could help. Thanks for the quick solution!

 

I will verify the solution in the real target app once I get through another Jython 2.7 b3 issue (see separate post).

 

Regards,

Johan

 

Van: Jeff Allen [[hidden email]]
Verzonden: woensdag 15 oktober 2014 0:19
Aan:
[hidden email]
Onderwerp: Re: [Jython-users] Unintuitive comparison between long and BigInteger

 

Johan:

I think this is now fixed under http://bugs.jython.org/issue2090, using the code contributed there. Thanks for helping us to understand it.

Santoso's observation is closely related, but not quite the same thing, because of differences in the way the JSR-223 and regular interpreters store and retrieve variables.

Jeff Allen

On 09/10/2014 23:44, Jeff Allen wrote:

This is very odd. Using essentially Johan's code:

TEST >>> a = 13L
None
TEST >>> repr(type(a))
<type 'java.math.BigInteger'>
TEST >>> repr(type(10L))
<type 'long'>

This is not to be confused with:
TEST >>> type(10L)
class org.python.core.PyLong

That happens, rather than <type 'long'>, because the output implicitly calls toString() instead or repr(), which a real console would.

It seems that values assigned to variables are stripped of their Jython wrapping. I believe this happens here:
https://hg.python.org/jython/file/1e8dd809df28/src/org/python/jsr223/PyScriptEngineScope.java#l147

The call of __tojava__ looks deliberate, but why? Sorry, it's late in the UK.

Jeff Allen

On 09/10/2014 21:23, Curtis Rueden wrote:

Hi Johan,

 

> This behavior doesn’t seem to occur in the default Jython console.

> Same instructions, correct result. As my simple example shows, this

> behavior does happen on Jython script evaluations through Java’s

> ScriptEngine interface.

 

Yep, it seems specific to org.python.jsr223.PyScriptEngine.

 

The reason the Fiji project I mentioned does not have the issue, even though it uses Jython in a JSR-223 context as well, is because that project uses the SciJava scripting framework's implementation [1] which wraps Jython's JSR-223 code _except_ for the ScriptEngine (probably because it predates Jython's official JSR-223 code). Specifically, the SciJava script engine evaluates Jython code as follows:

 

 

I tried tracing through the execution of your minimal script in the debugger in Eclipse, but I couldn't ultimately see what was going on, since the code gets compiled to Java bytecode before execution.

 

So I'm not sure what causes the difference, but if you need JSR-223 support and want to avoid this bug, a simple workaround would be to use the SciJava scripting framework's engine instead.

 

Regards,

Curtis


<big snip>

On Thu, Oct 9, 2014 at 1:19 PM, Santoso Wijaya <[hidden email]> wrote:

More succinctly, this:

 

Jython 2.7b3+ (default:2c45f75a5406, Oct 7 2014, 17:21:51) 

[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_67

Type "help", "copyright", "credits" or "license" for more information.

>>> from java.math import BigInteger

>>> a = BigInteger('10')

>>> type(a)

<type 'java.math.BigInteger'>

>>> a > 100

True

 


~/santoso

 

On Thu, Oct 9, 2014 at 2:24 AM, Johan Van Noten <[hidden email]> wrote:

Hi Jeff,

This issue keeps bugging me... also in 2.7-b3.
So I decided to make a small Java application to demonstrate the issue.

Some easy steps:
1) Download archive from http://www.qtc.be/TestJython.zip
2) Extract
3) Run the script TestJython.cmd (just runs a small class with the jython library on the cp)
4) Enter the example data (see below)
5) It is a small Eclipse Kepler project, so if you prefer, you can load/debug/run it from Eclipse.

When I do this, I get:
E:\tmp\Jython\TestJython>TestJython.cmd
TestJython >>> 10L > 100
false
TestJython >>> a = 10L
None
TestJython >>> a > 100
true
TestJython >>> type(a)
class java.math.BigInteger
TestJython >>> exit

So, as I confirmed initially, the underlying layers seem to use the BigInteger class, causing the failure in comparison.
As far as I know, I am not doing anything specific that could cause this.
I just use Jython as a Python Scripting Engine according to JSR-223, which is a convenient way of using it.
(Take a look at the extremely simple src/TestJython.java)

Do you expect this behavior?
Can it be avoided?

Johan

 

 


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho

_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users



------------------------------------------------------------------------------

_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: Unintuitive comparison between long and BigInteger

Johan Van Noten

Curtis,

 

Your suggestions sounds like logical behavior to me (being just a Jython user).

I wouldn’t know how to programmatically differentiate the “BigInteger by intent” objects from the “presented to Java as BigInteger but originally PyLongs”, though…

 

Johan

 

 

Van: [hidden email] [mailto:[hidden email]] Namens Curtis Rueden
Verzonden: donderdag 30 oktober 2014 19:07
Aan: Johan Van Noten
CC: Jeff Allen; [hidden email]
Onderwerp: Re: [Jython-users] Unintuitive comparison between long and BigInteger

 

Hi everyone,

 

Jeff Allen wrote:

> We now have, using Raphael's patch and an embedded JSR-233 script engine:

> TEST >>> from java.math import BigInteger

> TEST >>> a = BigInteger('10')

> TEST >>> type(a)

> class org.python.core.PyLong

 

Apologies if I'm being naive, but wouldn't the ultimately desired behavior be as follows?

 

TEST >>> from java.math import BigInteger

TEST >>> a = BigInteger('10')

TEST >>> type(a)

class java.math.BigInteger

TEST >>> a = 10L

TEST >>> type(a)

class org.python.core.PyLong

 

Otherwise, intuitively there might be problems in the "other direction" with legitimate BigIntegers being misinterpreted somehow, right?

 

The SciJava Jython script engine I mentioned earlier has the following behavior:

 

from java.math import BigInteger

a = BigInteger('10')

print(type(a))

a = 10L

print(type(a))

 

Produces:

 

<type 'java.math.BigInteger'>

<type 'long'>

 

Regards,

Curtis

(…)


------------------------------------------------------------------------------

_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
Reply | Threaded
Open this post in threaded view
|

Re: Unintuitive comparison between long and BigInteger

Jeff Allen-2
In reply to this post by Curtis Rueden
It's more of a jython-dev discussion, but yes, I suspect that would be
better.

Johan's test program is different from the Jython prompt in two ways. It
uses toString() to obtain what it prints, and it wraps a JSR-223
interpreter. The Jython prompt uses a repr() to obtain its result to
print, and a slightly different interpreter.

The toString() vs. repr() difference accounts for the way the type
object looks when we evaluate type(a), and a is a long, in Johan's test
program vs. at the prompt.

The relevant interpreter difference is in the container used as the
namespace. At the Jython prompt we use a container that holds the
objects as they are (like a dict). The JSR-223 interpreter uses a
container that strips the Jython wrapper, when storing objects, and
wraps a stored "naked" Java type into its corresponding Jython type.
Hence (since the patch) BigInteger gets wrapped to PyLong, whether it
started that way or not. type(10L) directly will give you a different
result from a = 10L followed by type(a) because the value is stored in
the namespace while the 10L is just on the stack.

I expect this unwrapping and wrapping is done to make namespaces more
friendly to Java code. I suspect the wrapping and unwrapping to be a
harmful divergence between the JSR-223 and regular interpreters. I'm not
sure what the ramifications would be of doing it differently.
(Presumably the namespace would have the "normal" implementation and the
unwrapping and wrapping would have to go on between it and Java.) This
seemed like a big change and to deserve more thought. But thanks for
raising it.

I was content to apply the patch because the class adapter from
BigInteger to PyLong ought anyway to exist in Jython. The JSR-223
interpreter, as designed, naturally uses it, which is at least
consistent with other types.

Jeff Allen

On 30/10/2014 18:07, Curtis Rueden wrote:

> Hi everyone,
>
> Jeff Allen wrote:
> > We now have, using Raphael's patch and an embedded JSR-233 script
> engine:
> > TEST >>> from java.math import BigInteger
> > TEST >>> a = BigInteger('10')
> > TEST >>> type(a)
> > class org.python.core.PyLong
>
> Apologies if I'm being naive, but wouldn't the ultimately desired
> behavior be as follows?
>
> TEST >>> from java.math import BigInteger
> TEST >>> a = BigInteger('10')
> TEST >>> type(a)
> class java.math.BigInteger
> TEST >>> a = 10L
> TEST >>> type(a)
> class org.python.core.PyLong
>
> Otherwise, intuitively there might be problems in the "other
> direction" with legitimate BigIntegers being misinterpreted somehow,
> right?
>
> The SciJava Jython script engine I mentioned earlier has the following
> behavior:
>
> from java.math import BigInteger
> a = BigInteger('10')
> print(type(a))
> a = 10L
> print(type(a))
>
> Produces:
>
> <type 'java.math.BigInteger'>
> <type 'long'>
>
> Regards,
> Curtis
>


------------------------------------------------------------------------------
_______________________________________________
Jython-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jython-users
12