[issue14700] Integer overflow in classic string formatting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor

New submission from Serhiy Storchaka <[hidden email]>:

Check for integer overflow for width and precision is buggy.

Just a few examples (on platform with 32-bit int):

>>> '%.21d' % 123
'000000000000000000123'
>>> '%.2147483648d' % 123
'123'
>>> '%.2147483650d' % 123
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: prec too big

>>> '%.21f' % (1./7)
'0.142857142857142849213'
>>> '%.2147483648f' % (1./7)
'0.142857'
>>> '%.2147483650f' % (1./7)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: prec too big

----------
components: Interpreter Core
messages: 159707
nosy: storchaka
priority: normal
severity: normal
status: open
title: Integer overflow in classic string formatting
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor

R. David Murray <[hidden email]> added the comment:

Serhiy: FYI we use the versions field to indicate which versions the fix will be made in, not which versions the bug occurs in.  Since only 2.7, 3.2, and 3.3 get bug fixes, I've changed the versions field to be just those three.  (3.1 and 2.6 are still in the list because they get *security* fixes, but those are rare.)

----------
nosy: +eric.smith, mark.dickinson, r.david.murray
versions:  -Python 2.6, Python 3.1

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

Indeed, Objects/unicodeobject.c (default branch) has this, at around line 13839:

                        if ((prec*10) / 10 != prec) {
                            PyErr_SetString(PyExc_ValueError,
                                            "prec too big");
                            goto onError;
                        }

... which since 'prec' has type int, will invoke undefined behaviour.  There are probably many other cases like this one.

Serhiy, what platform are you on?  And are you applying any special compile-time flags?  For gcc, we should be using -fwrapv, which in this case should make the above code work as intended.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

See get_integer in Objects/stringlib/unicode_format.h for a better way to do this sort of thing.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> Serhiy: FYI we use the versions field to indicate which versions the fix will be made in, not which versions the bug occurs in.  Since only 2.7, 3.2, and 3.3 get bug fixes, I've changed the versions field to be just those three.  (3.1 and 2.6 are still in the list because they get *security* fixes, but those are rare.)

Well, David, I understand. This ridiculous bug is unlikely security
issue.

Here is a patch that fixes this bug.

----------
keywords: +patch
Added file: http://bugs.python.org/file25426/pyunicode_format_integer_overflow.patch

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com


pyunicode_format_integer_overflow.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> Serhiy, what platform are you on?

32-bit Linux (Ubuntu), gcc 4.6. But it has to happen on any platform
with a 32-bit integer (for 64-bit use 9223372036854775808).

214748364*10/10 == 214748364 -- test passed
214748364*10 + ('8'-'0') == -2147483648 -- oops!

See also how is this problem solved in _struct.c.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

> But it has to happen on any platform
> with a 32-bit integer

Not necessarily:  it's undefined behaviour, so the compiler can do as it wishes.

Your patch should also address possible overflow of the addition.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> Your patch should also address possible overflow of the addition.

Here there is no overflow. The patch limits prec of a little stronger
(instead of 2147483647 to 2147483639 on a 32-bit platform).

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

Ah yes, true.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

Any chance of some tests? :-)

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> Any chance of some tests? :-)

Even a test for struct tests only struct.calcsize on this specific case.
For string formatting has no such function, on most platforms testing
would be a memory overflow.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> 32-bit Linux (Ubuntu), gcc 4.6.

Sorry, gcc 4.4.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor
In reply to this post by STINNER Victor

Mark Dickinson <[hidden email]> added the comment:

Still, I think it would be useful to have some tests that exercise the overflow branches.  (If those tests had existed before, then this issue would probably already have been found and fixed, since clang could have detected the undefined behaviour resulting from signed overflow.)

I'll add tests and apply this later.

----------
assignee:  -> mark.dickinson

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[issue14700] Integer overflow in classic string formatting

STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

> I'll add tests and apply this later.

Well, look at test_crasher in Lib/test/test_struct.py.

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue14700>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Loading...