PEP 418 is too divisive and confusing and should be postponed
Locked
123
123
PEP 418 is too divisive and confusing and should be postponed
 
|
Judging by the hundreds of emails regarding PEP 418, the disagreements about
APIs, namings, and even what characteristics clocks should have, I believe that the suggestion is too divisive (and confusing!) to be accepted or rejected at this time. Everyone has a different opinion, everyone believes their opinion holds for the majority, and it isn't practical for anyone to read the entire discussion. I propose for Python 3.3: 1) the os module should expose lightweight wrappers around whatever clocks the operating system provides; 2) the time module should NOT provide any further clocks other than the existing time() and clock() functions (but see point 4 below); 3) we postpone PEP 418 until there is some real-world experience with using the os clocks from Python and we can develop a consensus of what is actually needed rather than what people think we need (i.e. probably in 3.4); 4) if the standard library has need for a "use the best clock available, for some definition of best, and fall back to time() if not" clock, then the time module should do the simplest thing that could possible work, flagged as a private function: try: from os import bestclock as _bestclock except ImportError: _bestclock = time This can always be promoted to a public function later, if necessary. Python has worked pretty well without high res and monotonic clocks for 20 years. Let's not rush into a suboptimal design based on who can outlast everyone else in this discussion. -- Steven _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
On Wed, Apr 4, 2012 at 10:33 AM, Steven D'Aprano <[hidden email]> wrote:
> Python has worked pretty well without high res and monotonic clocks for 20 > years. Let's not rush into a suboptimal design based on who can outlast > everyone else in this discussion. +1 FWIW, I'd be fine with underscore prefixes on *any* additions to the relevant module APIs for 3.3. Cheers, Nick. -- Nick Coghlan | [hidden email] | Brisbane, Australia _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Steven D'Aprano-8
Le 04/04/2012 02:33, Steven D'Aprano a écrit :
> Judging by the hundreds of emails regarding PEP 418, the disagreements > about APIs, namings, and even what characteristics clocks should have, I > believe that the suggestion is too divisive (and confusing!) to be > accepted or rejected at this time. Oh, I just "rewrote" the PEP before reading your email. Sorry for the noise with this PEP :-) I just read again all emails related to this PEP to complete the PEP. The PEP should now list all proposed API designs. I hope that I did not forget anything. I failed to propose a consistent and clear API because I (and Guido!) wanted to fallback to the system clock. Falling back to the system clock is a problem when you have to define the function in the documentation or if you don't want to use the system clock (but do something else) if no monotonic clock is available. So I rewrote the PEP to simplify it: * Don't fallback to system clock: time.monotonic() is always monotonic (cannot go backward), but it is not always available. You have to write a classic try/except ImportError which has a nice advantage: your program will work on Python older than 3.3 ;-) * Remove the time.perf_counter() function (it was called time.highres() before). "highres" notion was confusing. I only wrote the function to expose QueryPerformanceCounter (while it was already accessible using os.clock()). The function was not well defined. Another PEP should be written or at least the subject should be discussed after the PEP 418 (monotonic clock). * Rename time.steady() to time.monotonic(), again :-) > Everyone has a different opinion, everyone believes their opinion holds > for the majority, and it isn't practical for anyone to read the entire > discussion. I read most emails and I can say that: * There is a need for a monotonic clock * Most people prefer to handle explicitly the fallback if no monotonic clock is available * Most people don't want to call the new function "steady" because it stands for something different > I propose for Python 3.3: > > 1) the os module should expose lightweight wrappers around whatever > clocks the operating system provides; Python 3.3 has already time.clock_gettime() and time.clock_getres() with CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES. mach_absolute_time() and GetTickCount/GetTick64 are not available yet. > 3) we postpone PEP 418 until there is some real-world experience with > using the os clocks from Python and we can develop a consensus of what > is actually needed rather than what people think we need (i.e. probably > in 3.4); Many applications already implement their own "monotonic" clock". Some libraries provide also such clock for Python. On UNIX, it's always using clock_gettime(MONOTONIC). On Windows, it's sometimes GetTickCount, sometimes QueryPerformanceCounter. On Mac OS X, it's always mach_absolute_time(). I didn't find a library supporting Solaris. > 4) if the standard library has need for a "use the best clock available, > for some definition of best, and fall back to time() if not" clock, then > the time module should do the simplest thing that could possible work, > flagged as a private function: In the last version of my PEP, time.monotonic() is simply defined as "a monotonic clock (cannot go backward)". There is no more "... best ..." in its definition. Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Steven D'Aprano-8
On 2012-04-03, at 8:33 PM, Steven D'Aprano wrote:
> 1) the os module should expose lightweight wrappers around whatever clocks the operating system provides; +1. That should make it flexible enough to those who really need it. > 2) the time module should NOT provide any further clocks other than the existing time() and clock() functions (but see point 4 below); > > 3) we postpone PEP 418 until there is some real-world experience with using the os clocks from Python and we can develop a consensus of what is actually needed rather than what people think we need (i.e. probably in 3.4); > > 4) if the standard library has need for a "use the best clock available, for some definition of best, and fall back to time() if not" clock, then the time module should do the simplest thing that could possible work, flagged as a private function: +1 on overall idea too. - Yury _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Victor Stinner-3
Finally! We've come full circle. +1 for monotonic as just described by Victor. _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
Lock it in before the paint dries. On Apr 4, 2012 10:05 AM, "Matt Joiner" <[hidden email]> wrote:
_______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Victor Stinner-3
On Wed, Apr 04, 2012 at 03:28:34AM +0200, Victor Stinner wrote:
> Le 04/04/2012 02:33, Steven D'Aprano a écrit : > >Judging by the hundreds of emails regarding PEP 418, the disagreements > >about APIs, namings, and even what characteristics clocks should have, I > >believe that the suggestion is too divisive (and confusing!) to be > >accepted or rejected at this time. > > Oh, I just "rewrote" the PEP before reading your email. Sorry for the > noise with this PEP :-) I just read again all emails related to this PEP > to complete the PEP. The PEP should now list all proposed API designs. I > hope that I did not forget anything. I think the PEP is a good, important PEP, and thank you for being the PEP's champion. But in my opinion, this is too big to rush it and risk locking in a sub-standard API for the next decade or two. > >I propose for Python 3.3: > > > >1) the os module should expose lightweight wrappers around whatever > >clocks the operating system provides; > > Python 3.3 has already time.clock_gettime() and time.clock_getres() with > CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES. Why does it already have these things when the PEP is not accepted? (This is not a rhetorical question, perhaps there is a good reason why these have been added independently of the PEP.) If I remember correctly, Guido earlier mentioned that he only wanted to see one or two (I forget which) new clocks, and I see in 3.3.0a1 there are already at least five new clocks: monotonic or clock_gettime(CLOCK_MONOTONIC) # Are these the same thing? wallclock clock_gettime(CLOCK_PROCESS_CPUTIME_ID) clock_gettime(CLOCK_REALTIME) clock_gettime(CLOCK_THREAD_CPUTIME_ID) plus the old ways, time.time and time.clock. (Neither of which seems to have a clock-id.) > mach_absolute_time() and GetTickCount/GetTick64 are not available yet. That will make potentially 10 different clocks in the time module. It may be that, eventually, Python should support all these ten different clocks. (Personally, I doubt that the average Python programmer cares about the difference between time() and clock(), let alone another eight more.) But there's no rush. I think we should start by supporting OS-specific clocks in the os module, and then once we have some best-practice idioms, we can promote some of them to the time module. -- Steven _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
On Wed, 4 Apr 2012 18:09:40 +1000
Steven D'Aprano <[hidden email]> wrote: > > Python 3.3 has already time.clock_gettime() and time.clock_getres() with > > CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, CLOCK_HIGHRES. > > Why does it already have these things when the PEP is not accepted? > > (This is not a rhetorical question, perhaps there is a good reason why > these have been added independently of the PEP.) Because these are thin (low-level) wrappers around the corresponding POSIX APIs, so there is no reason not to add them. I know you were asking for such wrappers to be in the "os" module, but my understanding is that time-related functions should preferably go into the "time" module. "os" is already full of very diverse stuff, and documentation-wise it is better if time-related functions end up in a time-related module. Otherwise we'll end up having to cross-link manually, which is always cumbersome (for us) and less practical (for the reader). Regards Antoine. _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
>> Why does it already have these things when the PEP is not accepted?
>> ... >> (This is not a rhetorical question, perhaps there is a good reason why >> these have been added independently of the PEP.) time.clock_gettime() & friends were added by the issue #10278. The function was added before someone asked (me) to write a PEP. The need of a PEP came later, when time.wallclock() and time.monotonic() functions were added. > Because these are thin (low-level) wrappers around the corresponding > POSIX APIs, so there is no reason not to add them. time.clock_gettime() can be used for other purpose than a monotonic clock. For example, CLOCK_THREAD_CPUTIME_ID is the only available function to get the "Thread-specific CPU-time clock". It also gives access to CLOCK_MONOTONIC_RAW which is not used by the time.monotonic() function proposed in the PEP. Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Victor Stinner-3
> I failed to propose a consistent and clear API because I (and Guido!) wanted
> to fallback to the system clock. Falling back to the system clock is a > problem when you have to define the function in the documentation or if you > don't want to use the system clock (but do something else) if no monotonic > clock is available. Well, it was not only Guido and me. Nick Coghlan wrote: "However, I think Victor's right to point out that the *standard library* needs to have a fallback to maintain backwards compatibility if time.monotonic() isn't available, and it seems silly to implement the same fallback logic in every module where we manipulate timeouts." and "Since duplicating that logic in every module that handles timeouts would be silly, it makes sense to provide an obvious way to do it in the time module." Michael Foord wrote: "It is this always-having-to-manually-fallback-depending-on-os that I was hoping your new functionality would avoid. Is time.try_monotonic() suitable for this usecase?" The following functions / libraries fall back to the system clock if no monotonic clock is available: - QElapsedTimer class of the Qt library - g_get_monotonic_time() of the glib library - monotonic_clock library - AbsoluteTime.now (third-party Ruby library), "AbsoluteTime.monotonic?" tells if AbsoluteTime.now is monotonic Extract of the glib doc: "Otherwise, we make a best effort that probably involves returning the wall clock time (with at least microsecond accuracy, subject to the limitations of the OS kernel)." -- Only the python-monotonic-time fails with an OSError if no monotonic clock is available. System.nanoTime() of Java has few garantee: "Returns the current value of the most precise available system timer, in nanoseconds. This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. The value returned represents nanoseconds since some fixed but arbitrary time (perhaps in the future, so values may be negative)." I don't even know if it is monotonic, steady or has an high resolution. Note: Boost.Chrono.high_resolution_clock falls back to the system clock if no steady clock is available. (But the high-resolution clock idea was deferred, it's something different than a monotonic or steady clock.) Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
I am fine with the PEP as it is now (2012-04-04 15:34 GMT).
A question: Since the only monotonic clock that can be adjusted by NTP is Linux' CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always give a clock that isn't adjusted by NTP. That would however mean we wouldn't support monotonic clocks on systems that run a Linux that is older than mid-2008. Is this generally seen as a problem? //Lennart _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Victor Stinner-3
On 2012-04-03, at 9:28 PM, Victor Stinner wrote:
> In the last version of my PEP, time.monotonic() is simply defined as "a monotonic clock (cannot go backward)". There is no more "... best ..." in its definition. I like the last version of the PEP ;) - Yury _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Lennart Regebro-2
Lennart Regebro wrote:
> Since the only monotonic clock that can be adjusted by NTP is Linux' > CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always > give a clock that isn't adjusted by NTP. I thought we decided that NTP adjustment isn't an issue, because it's always gradual. -- Greg _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Victor Stinner-3
> I failed to propose a consistent and clear API because I (and Guido!) wanted
> to fallback to the system clock. Falling back to the system clock is a > problem when you have to define the function in the documentation or if you > don't want to use the system clock (but do something else) if no monotonic > clock is available. More details why it's hard to define such function and why I dropped it from the PEP. If someone wants to propose again such function ("monotonic or fallback to system" clock), two issues should be solved: - name of the function - description of the function At least, "monotonic" and "steady" names are not acceptable names for such function, even if the function has an optional "strict=False" or "fallback=True" parameter. By the way, someone complained that having a boolean parameter requires to create a new function if you want to call it without an argument (use a lambda function, functools.partial, or anything else). Example: get_time = lambda: try_monotonic(fallback=True) t = get_time() The description should give the least guarantees. If the function doesn't promise anything (or only a few weak properties), it is harder to decide which clock do you need for your use case: time.clock(), time.time(), time.monotonic(), time.<name of the monotonic-of-fallback function>, ... Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
|
|
Maybe I missed it, but did anyone ever give a reason why the fallback couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the system clock? (Given a suitable minimum delta.) That function appeared to me to provide a sufficiently monotonic clock for timeout purposes, if nothing else. _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
On 04Apr2012 22:23, PJ Eby <[hidden email]> wrote:
| On Apr 4, 2012 7:28 PM, "Victor Stinner" <[hidden email]> wrote: | > More details why it's hard to define such function and why I dropped | > it from the PEP. | > | > If someone wants to propose again such function ("monotonic or | > fallback to system" clock), two issues should be solved: | > | > - name of the function | > - description of the function | | Maybe I missed it, but did anyone ever give a reason why the fallback | couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the | system clock? (Given a suitable minimum delta.) That function appeared to | me to provide a sufficiently monotonic clock for timeout purposes, if | nothing else. It was pointed out (by Nick Coglan I think?) that if the system clock stepped backwards then a timeout would be extended by at least that long. For example, code that waited (by polling the synthetic clock) for 1s could easily wait an hour if the system clock stepped back that far. Probaby undesirable. I think synthetic clocks are an extra task; they will all have side effects of one kind of another. A system monotonic clock, by contrast, may have access to some clock hardware that doesn't step when the "main" system clock gets adjusted, and can stay monotonic. A synthetic clock without such access can't behave as nicely. If synthetic clocks get handed out as fallback there should be some way for the user to know, or a big glaring negative guarrentee in the docs and on platforms without a system monotonic clock you might get a clock with weird (but monotonic!) behaviours if you use the fallback. Cheers, -- Cameron Simpson <[hidden email]> DoD#743 http://www.cskk.ezoshosting.com/cs/ Tachyon: A gluon that's not completely dry. _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
On Thu, Apr 5, 2012 at 1:41 PM, Cameron Simpson <[hidden email]> wrote:
> It was pointed out (by Nick Coglan I think?) that if the system clock > stepped backwards then a timeout would be extended by at least that > long. Guido pointed it out (it was in a reply to me, though). Cheers, Nick. -- Nick Coghlan | [hidden email] | Brisbane, Australia _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by Gregory Ewing
On Thu, Apr 5, 2012 at 00:45, Greg Ewing <[hidden email]> wrote:
> Lennart Regebro wrote: >> >> Since the only monotonic clock that can be adjusted by NTP is Linux' >> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always >> give a clock that isn't adjusted by NTP. > > I thought we decided that NTP adjustment isn't an issue, because > it's always gradual. Well, in timings it is an issue, but perhaps not a big one. :-) In any case, which one we use will not change the API, so if it is decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in the future, once Linux < 2.6.26 (or whatever it was) is deemed unsupported. //Lennart _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
>>> Since the only monotonic clock that can be adjusted by NTP is Linux'
>>> CLOCK_MONOTONIC, if we avoid it, then time.monotonic() would always >>> give a clock that isn't adjusted by NTP. >> >> I thought we decided that NTP adjustment isn't an issue, because >> it's always gradual. > > Well, in timings it is an issue, but perhaps not a big one. :-) > In any case, which one we use will not change the API, so if it is > decided it is an issue, we can always more to CLOCK_MONOTONIC_RAW in > the future, once Linux < 2.6.26 (or whatever it was) is deemed > unsupported. I prefer to use CLOCK_MONOTONIC, not because it is also available for older Linux kernels, but because it is more reliable. Even if the underlying clock source is unstable (unstable frequency), a delta of two reads of the CLOCK_MONOTONIC clock is a result in *seconds*, whereas CLOCK_MONOTONIC_RAW may use an unit a little bit bigger or smaller than a second. time.monotonic() unit is the second, as written in its documentation. Linux is the OS providing the most reliable monotonic clock, why should you use a not reliable monotonic clock instead? NTP doesn't step CLOCK_MONOTONIC, it only slew it. http://www.python.org/dev/peps/pep-0418/#ntp-adjustment Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
Re: PEP 418 is too divisive and confusing and should be postponed
|
|
In reply to this post by PJ Eby
2012/4/5 PJ Eby <[hidden email]>:
>> More details why it's hard to define such function and why I dropped >> it from the PEP. >> >> If someone wants to propose again such function ("monotonic or >> fallback to system" clock), two issues should be solved: >> >> - name of the function >> - description of the function > > Maybe I missed it, but did anyone ever give a reason why the fallback > couldn't be to Steven D'Aprano's monotonic wrapper algorithm over the system > clock? (Given a suitable minimum delta.) That function appeared to me to > provide a sufficiently monotonic clock for timeout purposes, if nothing > else. Did you read the following section of the PEP? http://www.python.org/dev/peps/pep-0418/#working-around-operating-system-bugs Did I miss something? If yes, could you write a patch for the PEP please? Victor _______________________________________________ Python-Dev mailing list [hidden email] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com |
«
Return to Python - python-dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |