Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Victor Stinner-3
>> http://hg.python.org/cpython/rev/7355550d5357
>> changeset:   75850:7355550d5357
>> user:        Stefan Krah <[hidden email]>
>> date:        Wed Mar 21 18:25:23 2012 +0100
>> summary:
>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
>> up the decimal module. Performance gains of the new C implementation are
>> between 12x and 80x, depending on the application.

Congrats Stefan! And thanks for the huge chunk of code.

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
Victor Stinner <[hidden email]> wrote:
> >>   Issue #7652: Integrate the decimal floating point libmpdec library to speed
> >> up the decimal module. Performance gains of the new C implementation are
> >> between 12x and 80x, depending on the application.
>
> Congrats Stefan! And thanks for the huge chunk of code.

Thanks, much appreciated. I'll take the opportunity to thank you in
return for the gigantic amount of work you've done on Python in the
past year!


Stefan Krah


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Dirkjan Ochtman
In reply to this post by Victor Stinner-3
On Wed, Mar 21, 2012 at 23:22, Victor Stinner <[hidden email]> wrote:
>>> http://hg.python.org/cpython/rev/7355550d5357
>>> changeset:   75850:7355550d5357
>>> user:        Stefan Krah <[hidden email]>
>>> date:        Wed Mar 21 18:25:23 2012 +0100
>>> summary:
>>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
>>> up the decimal module. Performance gains of the new C implementation are
>>> between 12x and 80x, depending on the application.

As a Python user, this looks really cool, thanks!

As a packager, is the libmpdec library used elsewhere? For Gentoo, we
generally prefer to package libraries separately and have Python
depend on them. From the site, it seems like you more or less wrote
libmpdec for usage in Python, but if it's general-purpose and actually
used in other software, it would be nice if Python grew a configure
option to make it use the system libmpdec.

Cheers,

Dirkjan
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Georg Brandl-2
In reply to this post by Victor Stinner-3
On 21.03.2012 23:22, Victor Stinner wrote:

>>> http://hg.python.org/cpython/rev/7355550d5357
>>> changeset:   75850:7355550d5357
>>> user:        Stefan Krah <[hidden email]>
>>> date:        Wed Mar 21 18:25:23 2012 +0100
>>> summary:
>>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
>>> up the decimal module. Performance gains of the new C implementation are
>>> between 12x and 80x, depending on the application.
>
> Congrats Stefan! And thanks for the huge chunk of code.

Seconded.  This is the kind of stuff that will make 3.3 the most awesomest
3.x release ever (and hopefully convince people that it does make sense to
port)...

cheers,
Georg

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Amaury Forgeot d'Arc
2012/3/22 Georg Brandl <[hidden email]>:
>> Congrats Stefan! And thanks for the huge chunk of code.
>
> Seconded.  This is the kind of stuff that will make 3.3 the most awesomest
> 3.x release ever (and hopefully convince people that it does make sense to
> port)...

On the other hand, porting PyPy to 3.3 will be more work ;-)
Fortunately the libmpdec directory should be reusable as is.
Nice work!

--
Amaury Forgeot d'Arc
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
In reply to this post by Dirkjan Ochtman
Dirkjan Ochtman <[hidden email]> wrote:
> As a packager, is the libmpdec library used elsewhere? For Gentoo, we
> generally prefer to package libraries separately and have Python
> depend on them. From the site, it seems like you more or less wrote
> libmpdec for usage in Python, but if it's general-purpose and actually
> used in other software, it would be nice if Python grew a configure
> option to make it use the system libmpdec.

libmpdec was actually written before the module. It's general purpose
and it fully implements the specification. I'm only aware of in-house
usage.

Someone has tried to submit a libmpdec package to OpenSUSE, but it
was rejected with the claim that there already exists a package
with the name. I think the claim is false: There is a libmpcdec
package, where "cdec" presumably stands for "codec".


I'll add the --with-system-libmpdec option with the caveat that
changes will probably make it first into the libmpdec shipped
with Python, see also:

http://bugs.python.org/issue7652#msg155744


On the bright side, I don't expect many changes, since the specification
is stable.


Stefan Krah



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Dirkjan Ochtman
On Fri, Mar 23, 2012 at 09:26, Stefan Krah <[hidden email]> wrote:
> I'll add the --with-system-libmpdec option with the caveat that
> changes will probably make it first into the libmpdec shipped
> with Python, see also:
>
> http://bugs.python.org/issue7652#msg155744

Sounds good, thanks!

Cheers,

Dirkjan
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
In reply to this post by Georg Brandl-2
Georg Brandl <[hidden email]> wrote:
> >>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
> >>> up the decimal module. Performance gains of the new C implementation are
> >>> between 12x and 80x, depending on the application.
> >
> > Congrats Stefan! And thanks for the huge chunk of code.
>
> Seconded.  This is the kind of stuff that will make 3.3 the most awesomest
> 3.x release ever (and hopefully convince people that it does make sense to
> port)...

Thanks! For cdecimal specifically I have the impression that 3.x is already
used in the financial community, where web framework dependencies aren't an
issue.

On the web side, there seems to be a huge interest in speeding up database
accesses, so let me evangelize again: Database applications using decimal
will run 12x faster in 3.3.


Stefan Krah


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Victor Stinner-3
By the way, how much faster is cdecimal? 72x or 80x?
http://docs.python.org/dev/whatsnew/3.3.html#decimal

Victor

2012/3/23 Stefan Krah <[hidden email]>:

> Georg Brandl <[hidden email]> wrote:
>> >>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
>> >>> up the decimal module. Performance gains of the new C implementation are
>> >>> between 12x and 80x, depending on the application.
>> >
>> > Congrats Stefan! And thanks for the huge chunk of code.
>>
>> Seconded.  This is the kind of stuff that will make 3.3 the most awesomest
>> 3.x release ever (and hopefully convince people that it does make sense to
>> port)...
>
> Thanks! For cdecimal specifically I have the impression that 3.x is already
> used in the financial community, where web framework dependencies aren't an
> issue.
>
> On the web side, there seems to be a huge interest in speeding up database
> accesses, so let me evangelize again: Database applications using decimal
> will run 12x faster in 3.3.
>
>
> Stefan Krah
>
>
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
In reply to this post by Amaury Forgeot d'Arc
Amaury Forgeot d'Arc <[hidden email]> wrote:
> > Seconded. ?This is the kind of stuff that will make 3.3 the most awesomest
> > 3.x release ever (and hopefully convince people that it does make sense to
> > port)...
>
> On the other hand, porting PyPy to 3.3 will be more work ;-)

We've got to keep you on your toes, don't we? :)


> Fortunately the libmpdec directory should be reusable as is.
> Nice work!

Thanks, also for helping out with the MutableMapping context.


Stefan Krah


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
In reply to this post by Victor Stinner-3
Victor Stinner <[hidden email]> wrote:
> By the way, how much faster is cdecimal? 72x or 80x?
> http://docs.python.org/dev/whatsnew/3.3.html#decimal

It really depends on the precision. Also, the performance of decimal.py
depends on many other things in the Python tree, so it easily changes
+-10%. Currently, decimal.py seems to be 10% faster than in 3.2, maybe
because of the new string representation.


The 80x is a ballpark figure for the maximum expected speedup for
standard numerical floating point applications.


factorial(1000) is 219x faster in _decimal, and with increasing
precision the difference gets larger and larger.


For huge numbers _decimal is also faster than int:

factorial(1000000):

_decimal, calculation time: 6.844487905502319
_decimal, tostr():          0.033592939376831055

int, calculation time: 17.96010398864746
int, tostr(): ... still running ...



Stefan Krah


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Antoine Pitrou
In reply to this post by Stefan Krah-3
On Fri, 23 Mar 2012 10:22:55 +0100
Stefan Krah <[hidden email]> wrote:

> Georg Brandl <[hidden email]> wrote:
> > >>>  Issue #7652: Integrate the decimal floating point libmpdec library to speed
> > >>> up the decimal module. Performance gains of the new C implementation are
> > >>> between 12x and 80x, depending on the application.
> > >
> > > Congrats Stefan! And thanks for the huge chunk of code.
> >
> > Seconded.  This is the kind of stuff that will make 3.3 the most awesomest
> > 3.x release ever (and hopefully convince people that it does make sense to
> > port)...
>
> Thanks! For cdecimal specifically I have the impression that 3.x is already
> used in the financial community, where web framework dependencies aren't an
> issue.
>
> On the web side, there seems to be a huge interest in speeding up database
> accesses, so let me evangelize again: Database applications using decimal
> will run 12x faster in 3.3.

Are you sure it isn't 12.5x ?

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
Antoine Pitrou <[hidden email]> wrote:
> > On the web side, there seems to be a huge interest in speeding up database
> > accesses, so let me evangelize again: Database applications using decimal
> > will run 12x faster in 3.3.
>
> Are you sure it isn't 12.5x ?

Well, that was marketing for 3.3.


Stefan Krah


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Victor Stinner-3
In reply to this post by Stefan Krah-3
> The 80x is a ballpark figure for the maximum expected speedup for
> standard numerical floating point applications.

Ok, but it's just surprising when you read the What's New document.
72x and 80x look to be inconsistent.

> For huge numbers _decimal is also faster than int:
>
> factorial(1000000):
>
> _decimal, calculation time: 6.844487905502319
> _decimal, tostr():          0.033592939376831055
>
> int, calculation time: 17.96010398864746
> int, tostr(): ... still running ...

Hum, with a resolution able to store the result with all digits? If
yes, would it be possible to reuse the multiply algorithm of _decimal
(and maybe of other functions) for int? Or does it depend heavily on
_decimal internal structures?

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: [Python-checkins] cpython: Issue #7652: Integrate the decimal floating point libmpdec library to speed

Stefan Krah-3
Victor Stinner <[hidden email]> wrote:
> > The 80x is a ballpark figure for the maximum expected speedup for
> > standard numerical floating point applications.
>
> Ok, but it's just surprising when you read the What's New document.
> 72x and 80x look to be inconsistent.

Yes, indeed, I'll reword that.


> > For huge numbers _decimal is also faster than int:
> >
> > factorial(1000000):
> >
> > _decimal, calculation time: 6.844487905502319
> > _decimal, tostr(): ?? ?? ?? ?? ??0.033592939376831055
> >
> > int, calculation time: 17.96010398864746
> > int, tostr(): ... still running ...
>
> Hum, with a resolution able to store the result with all digits?

Yes, you have to set context.prec (and emax) to the maximum values,
then the result is an exact integer. The conversion to string is
so fast because there is no complicated base conversion.


> If yes, would it be possible to reuse the multiply algorithm of _decimal
> (and maybe of other functions) for int? Or does it depend heavily on
> _decimal internal structures?

Large parts of the Number Theoretic Transform could be reused, but there
would be still quite a bit of work.


Stefan Krah



_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com