Removing surplus fields from the frame object and not adding any new ones.

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Removing surplus fields from the frame object and not adding any new ones.

Mark Shannon-3
The frame object is a key object in CPython. It holds the state
of a function invocation. Frame objects are allocated, initialised
and deallocated at a rapid rate.
Each extra field in the frame object requires extra work for each
and every function invocation. Fewer fields in the frame object
means less overhead for function calls, and cleaner simpler code.

We have recently removed the f_yieldfrom field from the frame object.
(http://bugs.python.org/issue14230)

The f_exc_type, f->f_exc_value, f->f_exc_traceback fields which handle
sys.exc_info() in generators could be moved to the generator object.
(http://bugs.python.org/issue13897)

The f_tstate field is redundant and, it would seem, dangerous
(http://bugs.python.org/issue14432)

The f_builtins, f_globals, f_locals fields could be combined into a
single f_namespaces struct.
(http://code.activestate.com/lists/python-dev/113381/)

Now PEP 419 proposes adding (yet) another field to the frame object.
Please don't.

Clean, concise data structures lead to clean, concise code.
which we all know is a "good thing" :)

Cheers,
Mark.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Andrew Svetlov
Do you want to create `frame` and `f_namespaces` every function call
instead of single `frame` creation?

On Mon, Apr 9, 2012 at 11:56 AM, Mark Shannon <[hidden email]> wrote:

> The frame object is a key object in CPython. It holds the state
> of a function invocation. Frame objects are allocated, initialised
> and deallocated at a rapid rate.
> Each extra field in the frame object requires extra work for each
> and every function invocation. Fewer fields in the frame object
> means less overhead for function calls, and cleaner simpler code.
>
> We have recently removed the f_yieldfrom field from the frame object.
> (http://bugs.python.org/issue14230)
>
> The f_exc_type, f->f_exc_value, f->f_exc_traceback fields which handle
> sys.exc_info() in generators could be moved to the generator object.
> (http://bugs.python.org/issue13897)
>
> The f_tstate field is redundant and, it would seem, dangerous
> (http://bugs.python.org/issue14432)
>
> The f_builtins, f_globals, f_locals fields could be combined into a
> single f_namespaces struct.
> (http://code.activestate.com/lists/python-dev/113381/)
>
> Now PEP 419 proposes adding (yet) another field to the frame object.
> Please don't.
>
> Clean, concise data structures lead to clean, concise code.
> which we all know is a "good thing" :)
>
> Cheers,
> Mark.
>
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



--
Thanks,
Andrew Svetlov
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Mark Shannon-3
Andrew Svetlov wrote:
> Do you want to create `frame` and `f_namespaces` every function call
> instead of single `frame` creation?

f_namespaces would be part of the frame, replacing f_builtins, f_globals
and f_locals. The indirection of an external object hurts performance,
so it would have to be a struct within the frame. The aim is clarity;
locals, globals and builtins form a trio, so should be implemented as such.

> On Mon, Apr 9, 2012 at 11:56 AM, Mark Shannon <[hidden email]> wrote:
>> The frame object is a key object in CPython. It holds the state
>> of a function invocation. Frame objects are allocated, initialised
>> and deallocated at a rapid rate.
>> Each extra field in the frame object requires extra work for each
>> and every function invocation. Fewer fields in the frame object
>> means less overhead for function calls, and cleaner simpler code.
>>
>> We have recently removed the f_yieldfrom field from the frame object.
>> (http://bugs.python.org/issue14230)
>>
>> The f_exc_type, f->f_exc_value, f->f_exc_traceback fields which handle
>> sys.exc_info() in generators could be moved to the generator object.
>> (http://bugs.python.org/issue13897)
>>
>> The f_tstate field is redundant and, it would seem, dangerous
>> (http://bugs.python.org/issue14432)
>>
>> The f_builtins, f_globals, f_locals fields could be combined into a
>> single f_namespaces struct.
>> (http://code.activestate.com/lists/python-dev/113381/)
>>
>> Now PEP 419 proposes adding (yet) another field to the frame object.
>> Please don't.
>>
>> Clean, concise data structures lead to clean, concise code.
>> which we all know is a "good thing" :)
>>
>> Cheers,
>> Mark.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> [hidden email]
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>
>
>

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Change to yield-from implementation

Gregory Ewing
In reply to this post by Mark Shannon-3
Mark Shannon wrote:

> We have recently removed the f_yieldfrom field from the frame object.
> (http://bugs.python.org/issue14230)

Hey, wait a minute. Did anyone consider the performance effect
of that change on deeply nested yield-froms?

The way it was, a yield-from chain was traversed by a very
tight C loop that found the end frame and resumed it directly.
If I understand what you've done correctly, now it has to
enter and execute a bytecode in every frame along the way.

--
Greg
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Change to yield-from implementation

Antoine Pitrou
On Tue, 10 Apr 2012 00:24:07 +1200
Greg Ewing <[hidden email]> wrote:
> Mark Shannon wrote:
>
> > We have recently removed the f_yieldfrom field from the frame object.
> > (http://bugs.python.org/issue14230)
>
> Hey, wait a minute. Did anyone consider the performance effect
> of that change on deeply nested yield-froms?

What's the point? Apart from naïve toy examples of traversing trees, I
don't think "deeply nested yield-froms" are likely to be
performance-critical.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Change to yield-from implementation

Benjamin Peterson-3
In reply to this post by Gregory Ewing
2012/4/9 Greg Ewing <[hidden email]>:

> Mark Shannon wrote:
>
>> We have recently removed the f_yieldfrom field from the frame object.
>> (http://bugs.python.org/issue14230)
>
>
> Hey, wait a minute. Did anyone consider the performance effect
> of that change on deeply nested yield-froms?
>
> The way it was, a yield-from chain was traversed by a very
> tight C loop that found the end frame and resumed it directly.
> If I understand what you've done correctly, now it has to
> enter and execute a bytecode in every frame along the way.

I think correctness is more important that performance, though.



--
Regards,
Benjamin
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Andrew Svetlov
In reply to this post by Mark Shannon-3
So it's really no difference between three separate fields in frame
and embedded struct with those fields.

On Mon, Apr 9, 2012 at 1:51 PM, Mark Shannon <[hidden email]> wrote:

> Andrew Svetlov wrote:
>>
>> Do you want to create `frame` and `f_namespaces` every function call
>> instead of single `frame` creation?
>
>
> f_namespaces would be part of the frame, replacing f_builtins, f_globals
> and f_locals. The indirection of an external object hurts performance,
> so it would have to be a struct within the frame. The aim is clarity;
> locals, globals and builtins form a trio, so should be implemented as such.
>
>
>> On Mon, Apr 9, 2012 at 11:56 AM, Mark Shannon <[hidden email]> wrote:
>>>
>>> The frame object is a key object in CPython. It holds the state
>>> of a function invocation. Frame objects are allocated, initialised
>>> and deallocated at a rapid rate.
>>> Each extra field in the frame object requires extra work for each
>>> and every function invocation. Fewer fields in the frame object
>>> means less overhead for function calls, and cleaner simpler code.
>>>
>>> We have recently removed the f_yieldfrom field from the frame object.
>>> (http://bugs.python.org/issue14230)
>>>
>>> The f_exc_type, f->f_exc_value, f->f_exc_traceback fields which handle
>>> sys.exc_info() in generators could be moved to the generator object.
>>> (http://bugs.python.org/issue13897)
>>>
>>> The f_tstate field is redundant and, it would seem, dangerous
>>> (http://bugs.python.org/issue14432)
>>>
>>> The f_builtins, f_globals, f_locals fields could be combined into a
>>> single f_namespaces struct.
>>> (http://code.activestate.com/lists/python-dev/113381/)
>>>
>>> Now PEP 419 proposes adding (yet) another field to the frame object.
>>> Please don't.
>>>
>>> Clean, concise data structures lead to clean, concise code.
>>> which we all know is a "good thing" :)
>>>
>>> Cheers,
>>> Mark.
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> [hidden email]
>>> http://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>>
>>> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>
>>
>>
>>
>
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



--
Thanks,
Andrew Svetlov
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Andrew Svetlov
While I agree with keeping data structures simple and clean I think
conserving them forever is bad idea in general.
Let's look on every particular case before making decision.

On Mon, Apr 9, 2012 at 3:46 PM, Andrew Svetlov <[hidden email]> wrote:

> So it's really no difference between three separate fields in frame
> and embedded struct with those fields.
>
> On Mon, Apr 9, 2012 at 1:51 PM, Mark Shannon <[hidden email]> wrote:
>> Andrew Svetlov wrote:
>>>
>>> Do you want to create `frame` and `f_namespaces` every function call
>>> instead of single `frame` creation?
>>
>>
>> f_namespaces would be part of the frame, replacing f_builtins, f_globals
>> and f_locals. The indirection of an external object hurts performance,
>> so it would have to be a struct within the frame. The aim is clarity;
>> locals, globals and builtins form a trio, so should be implemented as such.
>>
>>
>>> On Mon, Apr 9, 2012 at 11:56 AM, Mark Shannon <[hidden email]> wrote:
>>>>
>>>> The frame object is a key object in CPython. It holds the state
>>>> of a function invocation. Frame objects are allocated, initialised
>>>> and deallocated at a rapid rate.
>>>> Each extra field in the frame object requires extra work for each
>>>> and every function invocation. Fewer fields in the frame object
>>>> means less overhead for function calls, and cleaner simpler code.
>>>>
>>>> We have recently removed the f_yieldfrom field from the frame object.
>>>> (http://bugs.python.org/issue14230)
>>>>
>>>> The f_exc_type, f->f_exc_value, f->f_exc_traceback fields which handle
>>>> sys.exc_info() in generators could be moved to the generator object.
>>>> (http://bugs.python.org/issue13897)
>>>>
>>>> The f_tstate field is redundant and, it would seem, dangerous
>>>> (http://bugs.python.org/issue14432)
>>>>
>>>> The f_builtins, f_globals, f_locals fields could be combined into a
>>>> single f_namespaces struct.
>>>> (http://code.activestate.com/lists/python-dev/113381/)
>>>>
>>>> Now PEP 419 proposes adding (yet) another field to the frame object.
>>>> Please don't.
>>>>
>>>> Clean, concise data structures lead to clean, concise code.
>>>> which we all know is a "good thing" :)
>>>>
>>>> Cheers,
>>>> Mark.
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> [hidden email]
>>>> http://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe:
>>>>
>>>> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> [hidden email]
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>
>
>
> --
> Thanks,
> Andrew Svetlov



--
Thanks,
Andrew Svetlov
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Change to yield-from implementation

Guido van Rossum
In reply to this post by Antoine Pitrou
On Mon, Apr 9, 2012 at 5:46 AM, Antoine Pitrou <[hidden email]> wrote:

> On Tue, 10 Apr 2012 00:24:07 +1200
> Greg Ewing <[hidden email]> wrote:
>> Mark Shannon wrote:
>>
>> > We have recently removed the f_yieldfrom field from the frame object.
>> > (http://bugs.python.org/issue14230)
>>
>> Hey, wait a minute. Did anyone consider the performance effect
>> of that change on deeply nested yield-froms?
>
> What's the point? Apart from naïve toy examples of traversing trees, I
> don't think "deeply nested yield-froms" are likely to be
> performance-critical.

I agree with Benjamin that correctness trumps performance, but I'd
also like to point out that there are other use cases besides nested
iterators. If this gets used for coroutines it may not be so unusual
to have a stack of nested things with on top one that loops a lot --
if each iteration incurs cost proportional to how it got there this
may be a problem. But, correctness first.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Guido van Rossum
In reply to this post by Mark Shannon-3
On Mon, Apr 9, 2012 at 3:51 AM, Mark Shannon <[hidden email]> wrote:
> f_namespaces would be part of the frame, replacing f_builtins, f_globals
> and f_locals. The indirection of an external object hurts performance,
> so it would have to be a struct within the frame. The aim is clarity;
> locals, globals and builtins form a trio, so should be implemented as such.

How does replacing three fields with a struct containing three fields
reduce the size of the frame or the overhead in creating it?

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Mark Shannon-3
Guido van Rossum wrote:
> On Mon, Apr 9, 2012 at 3:51 AM, Mark Shannon <[hidden email]> wrote:
>> f_namespaces would be part of the frame, replacing f_builtins, f_globals
>> and f_locals. The indirection of an external object hurts performance,
>> so it would have to be a struct within the frame. The aim is clarity;
>> locals, globals and builtins form a trio, so should be implemented as such.
>
> How does replacing three fields with a struct containing three fields
> reduce the size of the frame or the overhead in creating it?
>

It doesn't.
I think it would improve clarity, but I doubt it is worth the effort.

The point I really wanted to make is that many of the fields in the
frame object belong elsewhere and adding new fields to the frame object
is generally a bad idea.

Cheers,
Mark.

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

"Martin v. Löwis"
> The point I really wanted to make is that many of the fields in the
> frame object belong elsewhere and adding new fields to the frame object
> is generally a bad idea.

I disagree with that statement, and don't think you have offered sufficient
proof of it. The structure may look irregular to you, but maybe you just need
to get used to it. Factually, I don't think that *many* of the fields belong
elsewhere. The majority of the fields clearly belongs where it is, and there
is nothing wrong with adding new fields if there is a need for it.

Regards,
Martin


_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|

Re: Removing surplus fields from the frame object and not adding any new ones.

Guido van Rossum
In reply to this post by Mark Shannon-3
On Mon, Apr 9, 2012 at 8:17 AM, Mark Shannon <[hidden email]> wrote:

> Guido van Rossum wrote:
>>
>> On Mon, Apr 9, 2012 at 3:51 AM, Mark Shannon <[hidden email]> wrote:
>>>
>>> f_namespaces would be part of the frame, replacing f_builtins, f_globals
>>> and f_locals. The indirection of an external object hurts performance,
>>> so it would have to be a struct within the frame. The aim is clarity;
>>> locals, globals and builtins form a trio, so should be implemented as
>>> such.
>>
>>
>> How does replacing three fields with a struct containing three fields
>> reduce the size of the frame or the overhead in creating it?
>>
>
> It doesn't.
> I think it would improve clarity, but I doubt it is worth the effort.
>
> The point I really wanted to make is that many of the fields in the
> frame object belong elsewhere and adding new fields to the frame object
> is generally a bad idea.

But is it? Consider the 'finally' proposal (not that I endorse it!) --
where would they put this info?

And what is the cost really? Have you measured it? Or are you just
optimizing prematurely?

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com