PEP 418 is too divisive and confusing and should be postponed

[Tim Delaney-2]

On 4 April 2012 10:33, Steven D'Aprano <[hidden email]> wrote:

try:
   from os import bestclock as _bestclock
except ImportError:
   _bestclock = time

My problem here is that "best clock" means different things to different people (as the number of emails shows).

I think exposing specific clocks is also useful (sometimes people may need a steady clock, and early failure is better than clock skew). However, I propose a loosely classified set of clocks built on top of the specific clocks, all of which can fall back to the lowest precision/non-monotonic clock if needed.

1. The "steadiest" clock on the system. Ideally this would be a steady clock, but may not be.

2. The "most precise" clock on the system. This would have the finest-grain tick available on the system.

3. The "highest performance" (or maybe "lowest latency") clock. This would be whichever clock on the system returned its results fastest.

I'm not sure if there are more that would be needed ("most accurate" comes to mind, but feels like it's arbitrarily choosing between steadiest and most precise, so I don't think it's valid).

By choosing relative terms, it caters to people's desire to have the "best" clock, but doesn't set the expectation that the behaviour is guaranteed.

Tim Delaney

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Victor Stinner-3]

> 1. The "steadiest" clock on the system. Ideally this would be a steady
> clock, but may not be.

time.monotonic() as proposed in the PEP 418 *is* the steadiest
available clock, but we cannot say that it is steady :-)

> 2. The "most precise" clock on the system. This would have the finest-grain
> tick available on the system.

It's discussed in the "Deferred API: time.perf_counter()" section. It
would be nice to provide such clock, but I don't feel able right now
to propose ab API for such requirement. It's unclear to me if it must
be monotonic, steady, count elapsed time during a sleep or not, etc.

It is already very hard to propose one single time function
(time.monotonic), so I chose to simplify the PEP and not propose two
functions but only one :-)

> 3. The "highest performance" (or maybe "lowest latency") clock. This would
> be whichever clock on the system returned its results fastest.

Linux provides CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE
clocks, reading the ACPI Power Management clock is known to be slow.
But should the clock be monotonic or not? Return seconds or CPU ticks?
If the clock is not well defined, it's useless or at least, not
portable. Exposing CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE
constants should be enough.

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Janzert-2]

On 4/5/2012 6:32 AM, Victor Stinner wrote:

>>>> Since the only monotonic clock that can be adjusted by NTP is Linux'
>>>> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always
>>>> give a clock that isn't adjusted by NTP.
>>>
>>> I thought we decided that NTP adjustment isn't an issue, because
>>> it's always gradual.
>>
>> Well, in timings it is an issue, but perhaps not a big one. :-)
>> In any case, which one we use will not change the API, so if it is
>> decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in
>> the future, once Linux<  2.6.26 (or whatever it was) is deemed
>> unsupported.
>
> I prefer to use CLOCK_MONOTONIC, not because it is also available for
> older Linux kernels, but because it is more reliable. Even if the
> underlying clock source is unstable (unstable frequency), a delta of
> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
> smaller than a second. time.monotonic() unit is the second, as written
> in its documentation.
>

I believe the above is only true for sufficiently large time deltas. One
of the major purposes of NTP slewing is to give up some short term
accuracy in order to achieve long term accuracy (e.g. whenever the clock
is found to be ahead of real time it is purposefully ticked slower than
real time).

So for benchmarking it would not be surprising to be better off with the
non-adjusted clock. Ideally there would be a clock that was slewed "just
enough" to try and achieve short term accuracy, but I don't know of
anything providing that.

Janzert

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Victor Stinner-3]

2012/4/7 Janzert <[hidden email]>:

> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>> older Linux kernels, but because it is more reliable. Even if the
>> underlying clock source is unstable (unstable frequency), a delta of
>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>> smaller than a second. time.monotonic() unit is the second, as written
>> in its documentation.
>
> I believe the above is only true for sufficiently large time deltas. One of
> the major purposes of NTP slewing is to give up some short term accuracy in
> order to achieve long term accuracy (e.g. whenever the clock is found to be
> ahead of real time it is purposefully ticked slower than real time).

I don't think that NTP works like that. NTP only uses very smooth adjustements:

""slewing": change the clock frequency to be slightly faster or slower
(which is done with adjtime()). Since the slew rate is limited to 0.5
ms/s, each second of adjustment requires an amortization interval of
2000 s. Thus, an adjustment of many seconds can take hours or days to
amortize."
http://www.python.org/dev/peps/pep-0418/#ntp-adjustment

> So for benchmarking it would not be surprising to be better off with the
> non-adjusted clock. Ideally there would be a clock that was slewed "just
> enough" to try and achieve short term accuracy, but I don't know of anything
> providing that.

time.monotonic() is not written for benchmarks. It does not have the
highest frequecency, it's primary property is that is monotonic. A
side effect is that it is usually the steadiest clock.

For example, on Windows time.monotonic() has only an accuracy of 15 ms
(15 milliseconds not 15 microseconds).

If you consider that the PEP should also solve the issue of
benchmarking clock, we should continue the work on this section:
http://www.python.org/dev/peps/pep-0418/#deferred-api-time-perf-counter

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Steven D'Aprano-8]

Victor Stinner wrote:

> 2012/4/7 Janzert <[hidden email]>:
>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>> older Linux kernels, but because it is more reliable. Even if the
>>> underlying clock source is unstable (unstable frequency), a delta of
>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>> smaller than a second. time.monotonic() unit is the second, as written
>>> in its documentation.
>> I believe the above is only true for sufficiently large time deltas. One of
>> the major purposes of NTP slewing is to give up some short term accuracy in
>> order to achieve long term accuracy (e.g. whenever the clock is found to be
>> ahead of real time it is purposefully ticked slower than real time).
>
> I don't think that NTP works like that. NTP only uses very smooth adjustements:
>
> ""slewing": change the clock frequency to be slightly faster or slower
> (which is done with adjtime()). Since the slew rate is limited to 0.5
> ms/s, each second of adjustment requires an amortization interval of
> 2000 s. Thus, an adjustment of many seconds can take hours or days to
> amortize."
> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment

That is incorrect. NTP by default will only slew the clock for small
discrepancies. For large discrepancies, it will step the clock, causing the
time to jump. By default, "large" here means more than 128 milliseconds.

Yes, milliseconds.

http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249


In any case, NTP is not the only thing that adjusts the clock, e.g. the
operating system will adjust the time for daylight savings.


--
Steven

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Victor Stinner-3]

2012/4/7 Steven D'Aprano <[hidden email]>:

> Victor Stinner wrote:
>>
>> 2012/4/7 Janzert <[hidden email]>:
>>>
>>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>>>
>>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>>> older Linux kernels, but because it is more reliable. Even if the
>>>> underlying clock source is unstable (unstable frequency), a delta of
>>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>>> smaller than a second. time.monotonic() unit is the second, as written
>>>> in its documentation.
>>>
>>> I believe the above is only true for sufficiently large time deltas. One
>>> of
>>> the major purposes of NTP slewing is to give up some short term accuracy
>>> in
>>> order to achieve long term accuracy (e.g. whenever the clock is found to
>>> be
>>> ahead of real time it is purposefully ticked slower than real time).
>>
>> I don't think that NTP works like that. NTP only uses very smooth
>> adjustements:
>>
>> ""slewing": change the clock frequency to be slightly faster or slower
>> (which is done with adjtime()). Since the slew rate is limited to 0.5
>> ms/s, each second of adjustment requires an amortization interval of
>> 2000 s. Thus, an adjustment of many seconds can take hours or days to
>> amortize."
>> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
>

> That is incorrect. NTP by default will only slew the clock for small
> discrepancies. For large discrepancies, it will step the clock, causing the
> time to jump. By default, "large" here means more than 128 milliseconds.
>
> Yes, milliseconds.
>
> http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249

We are talking about CLOCK_MONOTONIC. Steping is disabled on this clock.

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Cameron Simpson]

On 07Apr2012 20:40, Steven D'Aprano <[hidden email]> wrote:
| Victor Stinner wrote:
| > I don't think that NTP works like that. NTP only uses very smooth adjustements:
[...]
| > http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
|
| That is incorrect. NTP by default will only slew the clock for small
| discrepancies. For large discrepancies, it will step the clock, causing the
| time to jump. By default, "large" here means more than 128 milliseconds.
| Yes, milliseconds.
| http://www.ntp.org/ntpfaq/NTP-s-config-tricks.htm#AEN4249
|
| In any case, NTP is not the only thing that adjusts the clock, e.g. the
| operating system will adjust the time for daylight savings.

Ignoring the discussion of NTP, daylight saving is a _presentation_
issue. It is _display_. The OS clock does not change for daylight
saving! Think: "seconds since the epoch". This is a continuous function.
Daylight saving presentation occurs turning a seconds-since-epoch into a
human decomposed time (hours, etc).

Now, AFAIR, Windows used to run its system clock in "local time"
i.e. it did have to jump its clock for daylight saving. Hopefully that
is long gone.

UNIX never did this. It ran in seconds since the epoch (in its case, start of
01jan1970 GMT). Printing dates and timestamps to humans needed daylight
saving knowledge etc.

Cheers,
--
Cameron Simpson <[hidden email]> DoD#743
http://www.cskk.ezoshosting.com/cs/

Stan Malyshev <[hidden email]> wrote:
| You're dragging a peg in a blind hairpin when you see a patch of coolant
| up ahead, and you hear the airbrakes of an oncoming 18-wheeler.
| What do you do?  WHAT DO YOU DO?  ("Speed", anyone?)
Shoot my pillion?       - Vociferous Mole <[hidden email]>
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Glyph Lefkowitz]

On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:

In any case, NTP is not the only thing that adjusts the clock, e.g. the operating system will adjust the time for daylight savings.

Daylight savings time is not a clock adjustment, at least not in the sense this thread has mostly been talking about the word "clock".  It doesn't affect the "seconds from epoch" measurement, it affects the way in which the clock is formatted to the user.

-glyph

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Gregory P. Smith-3]

On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <[hidden email]> wrote:

On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:

In any case, NTP is not the only thing that adjusts the clock, e.g. the operating system will adjust the time for daylight savings.

Daylight savings time is not a clock adjustment, at least not in the sense this thread has mostly been talking about the word "clock".  It doesn't affect the "seconds from epoch" measurement, it affects the way in which the clock is formatted to the user.

-glyph

even on windows where the system hardware clock is maintained in local time?

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Guido van Rossum]

Is it still? I thought they fixed that ages ago?

On Mon, Apr 9, 2012 at 4:42 PM, Gregory P. Smith <[hidden email]> wrote:

>
> On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <[hidden email]>
> wrote:
>>
>> On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:
>>
>> In any case, NTP is not the only thing that adjusts the clock, e.g. the
>> operating system will adjust the time for daylight savings.
>>
>>
>> Daylight savings time is not a clock adjustment, at least not in the sense
>> this thread has mostly been talking about the word "clock".  It doesn't
>> affect the "seconds from epoch" measurement, it affects the way in which the
>> clock is formatted to the user.
>>
>> -glyph
>
>
> even on windows where the system hardware clock is maintained in local time?
>
>
> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Gregory P. Smith-3]

On Mon, Apr 9, 2012 at 4:46 PM, Guido van Rossum <[hidden email]> wrote:

Is it still? I thought they fixed that ages ago?

sadly, no.  http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html

On Mon, Apr 9, 2012 at 4:42 PM, Gregory P. Smith <[hidden email]> wrote:
>
> On Sat, Apr 7, 2012 at 4:56 PM, Glyph Lefkowitz <[hidden email]>
> wrote:
>>
>> On Apr 7, 2012, at 3:40 AM, Steven D'Aprano wrote:
>>
>> In any case, NTP is not the only thing that adjusts the clock, e.g. the
>> operating system will adjust the time for daylight savings.
>>
>>
>> Daylight savings time is not a clock adjustment, at least not in the sense
>> this thread has mostly been talking about the word "clock".  It doesn't
>> affect the "seconds from epoch" measurement, it affects the way in which the
>> clock is formatted to the user.
>>
>> -glyph
>
>
> even on windows where the system hardware clock is maintained in local time?
>
>

> _______________________________________________
> Python-Dev mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:

> http://mail.python.org/mailman/options/python-dev/guido%40python.org

>



--
--Guido van Rossum (python.org/~guido)

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Victor Stinner-3]

>> In any case, NTP is not the only thing that adjusts the clock, e.g. the
>> operating system will adjust the time for daylight savings.
>
> Daylight savings time is not a clock adjustment, at least not in the sense
> this thread has mostly been talking about the word "clock".  It doesn't
> affect the "seconds from epoch" measurement, it affects the way in which the
> clock is formatted to the user.

Ah yes, you're right. The system clock uses the UTC time zone on Linux
and Windows, and it is not affected by DST.

Victor
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com

[Janzert-2]

On 4/7/2012 5:49 AM, Victor Stinner wrote:

> 2012/4/7 Janzert<[hidden email]>:
>> On 4/5/2012 6:32 AM, Victor Stinner wrote:
>>> I prefer to use CLOCK_MONOTONIC, not because it is also available for
>>> older Linux kernels, but because it is more reliable. Even if the
>>> underlying clock source is unstable (unstable frequency), a delta of
>>> two reads of the CLOCK_MONOTONIC clock is a result in *seconds*,
>>> whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or
>>> smaller than a second. time.monotonic() unit is the second, as written
>>> in its documentation.
>>
>> I believe the above is only true for sufficiently large time deltas. One of
>> the major purposes of NTP slewing is to give up some short term accuracy in
>> order to achieve long term accuracy (e.g. whenever the clock is found to be
>> ahead of real time it is purposefully ticked slower than real time).
>
> I don't think that NTP works like that. NTP only uses very smooth adjustements:
>
> ""slewing": change the clock frequency to be slightly faster or slower
> (which is done with adjtime()). Since the slew rate is limited to 0.5
> ms/s, each second of adjustment requires an amortization interval of
> 2000 s. Thus, an adjustment of many seconds can take hours or days to
> amortize."
> http://www.python.org/dev/peps/pep-0418/#ntp-adjustment
>

Right, the description in that paragraph is exactly what I was referring
to above. :) It is unfortunate that a clock with a resolution of 1ns may
be purposefully thrown off by 500,000ns per second in the short term.

In practice you are probably correct that it is better to take the
slewed clock even though it may have purposeful short term inaccuracy
thrown in than it is to use the completely unadjusted one.

>> So for benchmarking it would not be surprising to be better off with the
>> non-adjusted clock. Ideally there would be a clock that was slewed "just
>> enough" to try and achieve short term accuracy, but I don't know of anything
>> providing that.
>
> time.monotonic() is not written for benchmarks. It does not have the
> highest frequecency, it's primary property is that is monotonic. A
> side effect is that it is usually the steadiest clock.
>
> For example, on Windows time.monotonic() has only an accuracy of 15 ms
> (15 milliseconds not 15 microseconds).
>

Hmm, I just realized an unfortunate result of that. Since it means
time.monotonic() will be too coarse to be useful for frame rate level
timing on windows.

Janzert

_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com