Quantcast

PEP 411 - request for pronouncement

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

PEP 411 - request for pronouncement

Eli Bendersky
Hello,

PEP 411 -- Provisional packages in the Python standard library

Has been updated with all accumulated feedback from list discussions.
Here it is: http://www.python.org/dev/peps/pep-0411/ (the text is also
pasted in the bottom of this email).

The PEP received mostly positive feedback. The only undecided point is
where to specify that the package is provisional. Currently the PEP
mandates to specify it in the documentation and in the docstring.
Other suggestions were to put it in the code, either as a
__provisional__ attribute on the module, or collect all such modules
in a single sys.provisional list.

According to http://blog.python.org/2012/03/2012-language-summit-report.html,
the PEP was discussed in the language summit and overall viewed
positively, although no final decision has been reached.

ISTM a decision needs to be taken, which is why I request
pronouncement, with a recommendation on the requirement the PEP should
make of provisional modules (process details).

Eli

--------------------------------

PEP: 411
Title: Provisional packages in the Python standard library
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <[hidden email]>,
        Eli Bendersky <[hidden email]>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 2012-02-10
Python-Version: 3.3
Post-History: 2012-02-10


Abstract
========

The process of including a new package into the Python standard library is
hindered by the API lock-in and promise of backward compatibility implied by
a package being formally part of Python.  This PEP describes a methodology
for marking a standard library package "provisional" for the period of a single
minor release.  A provisional package may have its API modified prior to
"graduating" into a "stable" state.  On one hand, this state provides the
package with the benefits of being formally part of the Python distribution.
On the other hand, the core development team explicitly states that no promises
are made with regards to the the stability of the package's API, which may
change for the next release.  While it is considered an unlikely outcome,
such packages may even be removed from the standard library without a
deprecation period if the concerns regarding their API or maintenance prove
well-founded.


Proposal - a documented provisional state
=========================================

Whenever the Python core development team decides that a new package should be
included into the standard library, but isn't entirely sure about whether the
package's API is optimal, the package can be included and marked as
"provisional".

In the next minor release, the package may either be "graduated" into a normal
"stable" state in the standard library, remain in provisional state, or be
rejected and removed entirely from the Python source tree.  If the package ends
up graduating into the stable state after being provisional, its API may
be changed according to accumulated feedback.  The core development team
explicitly makes no guarantees about API stability and backward compatibility
of provisional packages.


Marking a package provisional
-----------------------------

A package will be marked provisional by a notice in its documentation page and
its docstring. The following paragraph will be added as a note at the top of
the documentation page:

    The <X> package has been included in the standard library on a
    provisional basis.  Backwards incompatible changes (up to and including
    removal of the package) may occur if deemed necessary by the core
    developers.

The phrase "provisional basis" will then be a link to the glossary term
"provisional package", defined as:

    A provisional package is one which has been deliberately excluded from the
    standard library's normal backwards compatibility guarantees.  While major
    changes to such packages are not expected, as long as they are marked
    provisional, backwards incompatible changes (up to and including removal of
    the package) may occur if deemed necessary by core developers.  Such changes
    will not be made gratuitously - they will occur only if serious flaws are
    uncovered that were missed prior to the inclusion of the package.

    This process allows the standard library to continue to evolve over time,
    without locking in problematic design errors for extended periods of time.
    See PEP 411 for more details.

The following will be added to the start of the packages's docstring:

    The API of this package is currently provisional.  Refer to the
    documentation for details.

Moving a package from the provisional to the stable state simply implies
removing these notes from its documentation page and docstring.


Which packages should go through the provisional state
------------------------------------------------------

We expect most packages proposed for addition into the Python standard library
to go through a minor release in the provisional state. There may, however,
be some exceptions, such as packages that use a pre-defined API (for example
``lzma``, which generally follows the API of the existing ``bz2`` package),
or packages with an API that has wide acceptance in the Python development
community.

In any case, packages that are proposed to be added to the standard library,
whether via the provisional state or directly, must fulfill the acceptance
conditions set by PEP 2.

Criteria for "graduation"
-------------------------

In principle, most provisional packages should eventually graduate to the
stable standard library.  Some reasons for not graduating are:

* The package may prove to be unstable or fragile, without sufficient developer
  support to maintain it.
* A much better alternative package may be found during the preview release.

Essentially, the decision will be made by the core developers on a per-case
basis.  The point to emphasize here is that a package's inclusion in the
standard library as "provisional" in some release does not guarantee it will
continue being part of Python in the next release.


Rationale
=========

Benefits for the core development team
--------------------------------------

Currently, the core developers are really reluctant to add new interfaces to
the standard library.  This is because as soon as they're published in a
release, API design mistakes get locked in due to backward compatibility
concerns.

By gating all major API additions through some kind of a provisional mechanism
for a full release, we get one full release cycle of community feedback
before we lock in the APIs with our standard backward compatibility guarantee.

We can also start integrating provisional packages with the rest of the standard
library early, so long as we make it clear to packagers that the provisional
packages should not be considered optional.  The only difference between
provisional APIs and the rest of the standard library is that provisional APIs
are explicitly exempted from the usual backward compatibility guarantees.

Benefits for end users
----------------------

For future end users, the broadest benefit lies in a better "out-of-the-box"
experience - rather than being told "oh, the standard library tools for task X
are horrible, download this 3rd party library instead", those superior tools
are more likely to be just be an import away.

For environments where developers are required to conduct due diligence on
their upstream dependencies (severely harming the cost-effectiveness of, or
even ruling out entirely, much of the material on PyPI), the key benefit lies
in ensuring that all packages in the provisional state are clearly under
python-dev's aegis from at least the following perspectives:

* Licensing:  Redistributed by the PSF under a Contributor Licensing Agreement.
* Documentation: The documentation of the package is published and organized via
  the standard Python documentation tools (i.e. ReST source, output generated
  with Sphinx and published on http://docs.python.org).
* Testing: The package test suites are run on the python.org buildbot fleet
  and results published via http://www.python.org/dev/buildbot.
* Issue management: Bugs and feature requests are handled on
  http://bugs.python.org
* Source control: The master repository for the software is published
  on http://hg.python.org.


Candidates for provisional inclusion into the standard library
==============================================================

For Python 3.3, there are a number of clear current candidates:

* ``regex`` (http://pypi.python.org/pypi/regex) - approved by Guido [#]_.
* ``daemon`` (PEP 3143)
* ``ipaddr`` (PEP 3144)

Other possible future use cases include:

* Improved HTTP modules (e.g. ``requests``)
* HTML 5 parsing support (e.g. ``html5lib``)
* Improved URL/URI/IRI parsing
* A standard image API (PEP 368)
* Improved encapsulation of import state (PEP 406)
* Standard event loop API (PEP 3153)
* A binary version of WSGI for Python 3 (e.g. PEP 444)
* Generic function support (e.g. ``simplegeneric``)


Rejected alternatives and variations
====================================

See PEP 408.


References
==========

.. [#] http://mail.python.org/pipermail/python-dev/2012-January/115962.html

Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Lennart Regebro-2
On Fri, Mar 23, 2012 at 10:51, Eli Bendersky <[hidden email]> wrote:
> The PEP received mostly positive feedback. The only undecided point is
> where to specify that the package is provisional. Currently the PEP
> mandates to specify it in the documentation and in the docstring.
> Other suggestions were to put it in the code, either as a
> __provisional__ attribute on the module, or collect all such modules
> in a single sys.provisional list.

I'm not sure what the usecase is for checking in code if a module is
provisional or not. It doesn't seem useful, and risks being
unmaintained, especially when the flag is on the module itself.

//Lennart
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Eli Bendersky
On Sat, Mar 24, 2012 at 13:53, Lennart Regebro <[hidden email]> wrote:

> On Fri, Mar 23, 2012 at 10:51, Eli Bendersky <[hidden email]> wrote:
>> The PEP received mostly positive feedback. The only undecided point is
>> where to specify that the package is provisional. Currently the PEP
>> mandates to specify it in the documentation and in the docstring.
>> Other suggestions were to put it in the code, either as a
>> __provisional__ attribute on the module, or collect all such modules
>> in a single sys.provisional list.
>
> I'm not sure what the usecase is for checking in code if a module is
> provisional or not. It doesn't seem useful, and risks being
> unmaintained, especially when the flag is on the module itself.
>

Some usecases were given by Jim J. Jewett here:
http://mail.python.org/pipermail/python-dev/2012-February/116400.html
(and +1-ed by a couple of others.)

Eli
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Lennart Regebro-2
On Mon, Mar 26, 2012 at 05:40, Eli Bendersky <[hidden email]> wrote:

> On Sat, Mar 24, 2012 at 13:53, Lennart Regebro <[hidden email]> wrote:
>> On Fri, Mar 23, 2012 at 10:51, Eli Bendersky <[hidden email]> wrote:
>>> The PEP received mostly positive feedback. The only undecided point is
>>> where to specify that the package is provisional. Currently the PEP
>>> mandates to specify it in the documentation and in the docstring.
>>> Other suggestions were to put it in the code, either as a
>>> __provisional__ attribute on the module, or collect all such modules
>>> in a single sys.provisional list.
>>
>> I'm not sure what the usecase is for checking in code if a module is
>> provisional or not. It doesn't seem useful, and risks being
>> unmaintained, especially when the flag is on the module itself.
>>
>
> Some usecases were given by Jim J. Jewett here:
> http://mail.python.org/pipermail/python-dev/2012-February/116400.html
> (and +1-ed by a couple of others.)

Having a list of provisional modules seems the most helpful there, in
that case, and also noting that it is provisional in the module docs
shown when you do help().
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Guido van Rossum
In reply to this post by Eli Bendersky
On Fri, Mar 23, 2012 at 2:51 AM, Eli Bendersky <[hidden email]> wrote:

> PEP 411 -- Provisional packages in the Python standard library
>
> Has been updated with all accumulated feedback from list discussions.
> Here it is: http://www.python.org/dev/peps/pep-0411/ (the text is also
> pasted in the bottom of this email).
>
> The PEP received mostly positive feedback. The only undecided point is
> where to specify that the package is provisional. Currently the PEP
> mandates to specify it in the documentation and in the docstring.
> Other suggestions were to put it in the code, either as a
> __provisional__ attribute on the module, or collect all such modules
> in a single sys.provisional list.
>
> According to http://blog.python.org/2012/03/2012-language-summit-report.html,
> the PEP was discussed in the language summit and overall viewed
> positively, although no final decision has been reached.
>
> ISTM a decision needs to be taken, which is why I request
> pronouncement, with a recommendation on the requirement the PEP should
> make of provisional modules (process details).

I think the PEP is almost ready for approval. Congratulations! A few comments:

- I'd leave some wiggle room for the docs owner (Georg) about the
exact formulation of the text blurb included for provisional modules
and the glossary entry; I don't want the PEP to have the last word
here.

- I think we are settling on the term "feature release" instead of the
somewhat ambiguous "minor release".

- As was discussed at the language summit, I'd like to emphasize that
the bar for making changes to a provisional package should be
considered pretty high. That is, while we don't make guarantees about
backward compatibility, we still expect that most of the API of most
provisional packages will be unchanged at graduation. Withdrawals
should also be pretty rare.

- Should we limit the duration of the provisional state to 1 or 2
feature releases?

- I'm not sure what to do with regex -- it may be better to just
include in as "re" and keep the old re module around under another
name ("sre" has been proposed half jokingly).

PS. Please use the version in the peps repo as the starting point of
future edits.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Eli Bendersky
> I think the PEP is almost ready for approval. Congratulations! A few comments:
>
> - I'd leave some wiggle room for the docs owner (Georg) about the
> exact formulation of the text blurb included for provisional modules
> and the glossary entry; I don't want the PEP to have the last word
> here.

Sure, Georg is free to modify the pep to amend the formulation if he wants to.

>
> - I think we are settling on the term "feature release" instead of the
> somewhat ambiguous "minor release".

Fixed

>
> - As was discussed at the language summit, I'd like to emphasize that
> the bar for making changes to a provisional package should be
> considered pretty high. That is, while we don't make guarantees about
> backward compatibility, we still expect that most of the API of most
> provisional packages will be unchanged at graduation. Withdrawals
> should also be pretty rare.
>

Added this emphasis at the end of the "Criteria for graduation" section.

> - Should we limit the duration of the provisional state to 1 or 2
> feature releases?

Initially the PEP came out with a 1-release limit, but some of the
devs pointed out
(http://mail.python.org/pipermail/python-dev/2012-February/116406.html)
that me should not necessarily restrict ourselves.

>
> - I'm not sure what to do with regex -- it may be better to just
> include in as "re" and keep the old re module around under another
> name ("sre" has been proposed half jokingly).
>

Document updated in the PEPs Hg, rev a1bb0a9af63f.

Eli
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Guido van Rossum
On Mon, Mar 26, 2012 at 8:23 PM, Eli Bendersky <[hidden email]> wrote:
>> I think the PEP is almost ready for approval. Congratulations! A few comments:
>>
>> - I'd leave some wiggle room for the docs owner (Georg) about the
>> exact formulation of the text blurb included for provisional modules
>> and the glossary entry; I don't want the PEP to have the last word
>> here.
>
> Sure, Georg is free to modify the pep to amend the formulation if he wants to.

Ok. He can do that at any time. :-)

>> - I think we are settling on the term "feature release" instead of the
>> somewhat ambiguous "minor release".
>
> Fixed

Great.

>> - As was discussed at the language summit, I'd like to emphasize that
>> the bar for making changes to a provisional package should be
>> considered pretty high. That is, while we don't make guarantees about
>> backward compatibility, we still expect that most of the API of most
>> provisional packages will be unchanged at graduation. Withdrawals
>> should also be pretty rare.
>>
>
> Added this emphasis at the end of the "Criteria for graduation" section.

Cool.

>> - Should we limit the duration of the provisional state to 1 or 2
>> feature releases?
>
> Initially the PEP came out with a 1-release limit, but some of the
> devs pointed out
> (http://mail.python.org/pipermail/python-dev/2012-February/116406.html)
> that me should not necessarily restrict ourselves.

Gotcha.

>> - I'm not sure what to do with regex -- it may be better to just
>> include in as "re" and keep the old re module around under another
>> name ("sre" has been proposed half jokingly).
>>
>
> Document updated in the PEPs Hg, rev a1bb0a9af63f.

TBH I'm not sure what I meant to say about regex in
http://mail.python.org/pipermail/python-dev/2012-January/115962.html
... But I think if we end up changing the decision about this or any
others that doesn't invalidate this PEP, which is informational PEP
anyway.

I've marked it up as Approved. Thanks, and congrats!

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Eli Bendersky
>
> I've marked it up as Approved. Thanks, and congrats!
>

Thanks!

Eli
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PEP 411 - request for pronouncement

Eli Bendersky
In reply to this post by Guido van Rossum
On Tue, Mar 27, 2012 at 05:34, Guido van Rossum <[hidden email]> wrote:

> On Mon, Mar 26, 2012 at 8:23 PM, Eli Bendersky <[hidden email]> wrote:
>>> I think the PEP is almost ready for approval. Congratulations! A few comments:
>>>
>>> - I'd leave some wiggle room for the docs owner (Georg) about the
>>> exact formulation of the text blurb included for provisional modules
>>> and the glossary entry; I don't want the PEP to have the last word
>>> here.
>>
>> Sure, Georg is free to modify the pep to amend the formulation if he wants to.
>
> Ok. He can do that at any time. :-)
>

Georg, would you like to change the suggested phrasing in the PEP, or
can I go on and add the "provisional package" term to the glossary (in
3.3 only, of course).

Eli
_______________________________________________
Python-Dev mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%2B1324100855712-1801473%40n6.nabble.com
Loading...