Adding a "Pure Python" flag to PyPI

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding a "Pure Python" flag to PyPI

Paul Moore
I'm trying to identify which distributions in PyPI need to be made
available in binary format for people without C compilers (i.e.,
distributions including C extensions). I could use the classifier
information, but as that is user-supplied, it is not 100% reliable[1].
(Also, it's not in the data set I have available, although that's a
fixable issue). I'm only interested in existing binary distributions
(eggs and wininst/msi packages) on PyPI.

The problem is that simply assuming that because a distribution
provides binary installers, or even version-specific installers, is
not enough. For example, look at PyParsing - it provides a
version-specific wininst installer for every Python version, but it's
a pure Python package, and can easily be installed from source.

I can't see a way of reliably establishing whether a distribution is
"pure Python", and yet distutils/packaging clearly has that
information available when building. Would it be worthwhile adding a
"pure Python" flag to the PyPI classifiers, which could be
automatically populated by packaging? We'd still be reliant on people
who manually maintain metadata getting it correct, but it would help
in many cases (and in particular, in those cases where projects do
regularly upload binary distributions).

Alternatively, if there is a way of reliably identifying those
packages that can't be installed from source by someone without a
compiler, I'd be interested to know.

Thanks,
Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Fwd: Adding a "Pure Python" flag to PyPI

Paul Moore
[Resending because I don't think the first try made it to the list...]

I'm trying to identify which distributions in PyPI need to be made
available in binary format for people without C compilers (i.e.,
distributions including C extensions). I could use the classifier
information, but as that is user-supplied, it is not 100% reliable[1].
(Also, it's not in the data set I have available, although that's a
fixable issue). I'm only interested in existing binary distributions
(eggs and wininst/msi packages) on PyPI.

The problem is that simply assuming that because a distribution
provides binary installers, or even version-specific installers, is
not enough. For example, look at PyParsing - it provides a
version-specific wininst installer for every Python version, but it's
a pure Python package, and can easily be installed from source.

I can't see a way of reliably establishing whether a distribution is
"pure Python", and yet distutils/packaging clearly has that
information available when building. Would it be worthwhile adding a
"pure Python" flag to the PyPI classifiers, which could be
automatically populated by packaging? We'd still be reliant on people
who manually maintain metadata getting it correct, but it would help
in many cases (and in particular, in those cases where projects do
regularly upload binary distributions).

Alternatively, if there is a way of reliably identifying those
packages that can't be installed from source by someone without a
compiler, I'd be interested to know.

Thanks,
Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Adding a "Pure Python" flag to PyPI

Fuzzyman-2


On 5 March 2012 18:43, Paul Moore <[hidden email]> wrote:
[Resending because I don't think the first try made it to the list...]

I'm trying to identify which distributions in PyPI need to be made
available in binary format for people without C compilers (i.e.,
distributions including C extensions). I could use the classifier
information, but as that is user-supplied, it is not 100% reliable[1].
(Also, it's not in the data set I have available, although that's a
fixable issue). I'm only interested in existing binary distributions
(eggs and wininst/msi packages) on PyPI.

The problem is that simply assuming that because a distribution
provides binary installers, or even version-specific installers, is
not enough. For example, look at PyParsing - it provides a
version-specific wininst installer for every Python version, but it's
a pure Python package, and can easily be installed from source.

I can't see a way of reliably establishing whether a distribution is
"pure Python", and yet distutils/packaging clearly has that
information available when building. Would it be worthwhile adding a
"pure Python" flag to the PyPI classifiers, which could be
automatically populated by packaging? We'd still be reliant on people
who manually maintain metadata getting it correct, but it would help
in many cases (and in particular, in those cases where projects do
regularly upload binary distributions).

+1

It's useful information for users - if only source distributions are available for packages that require compilation some users can't use them (typically Windows users), so a pure-python flag would be helpful.

Michael

 

Alternatively, if there is a way of reliably identifying those
packages that can't be installed from source by someone without a
compiler, I'd be interested to know.

Thanks,
Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig



--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Adding a "Pure Python" flag to PyPI

"Martin v. Löwis"
In reply to this post by Paul Moore
> I can't see a way of reliably establishing whether a distribution is
> "pure Python", and yet distutils/packaging clearly has that
> information available when building. Would it be worthwhile adding a
> "pure Python" flag to the PyPI classifiers, which could be
> automatically populated by packaging? We'd still be reliant on people
> who manually maintain metadata getting it correct, but it would help
> in many cases (and in particular, in those cases where projects do
> regularly upload binary distributions).

I don't think it's worthwhile. It would take forever (literally decades)
for this to get into wide use, unless some tool enforces it (e.g. PyPI
refuses the upload if there is a C file in the source tarball, yet the
package is not marked pure C).

> Alternatively, if there is a way of reliably identifying those
> packages that can't be installed from source by someone without a
> compiler, I'd be interested to know.

Depends on how reliable you want it. Whatever mechanism someone can
propose, I can find a way to cheat that mechanism.

However, downloading the source distribution and inspecting it should
be fairly reliable.

Regards,
Martin
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Adding a "Pure Python" flag to PyPI

Jim Fulton
On Mon, Mar 5, 2012 at 4:21 PM, "Martin v. Löwis" <[hidden email]> wrote:

>> I can't see a way of reliably establishing whether a distribution is
>> "pure Python", and yet distutils/packaging clearly has that
>> information available when building. Would it be worthwhile adding a
>> "pure Python" flag to the PyPI classifiers, which could be
>> automatically populated by packaging? We'd still be reliant on people
>> who manually maintain metadata getting it correct, but it would help
>> in many cases (and in particular, in those cases where projects do
>> regularly upload binary distributions).
>
> I don't think it's worthwhile. It would take forever (literally decades)
> for this to get into wide use, unless some tool enforces it (e.g. PyPI
> refuses the upload if there is a C file in the source tarball, yet the
> package is not marked pure C).

distutils/2/packaging should be able to set this automatically.

If we do this, maybe the possible values should be yes, no, and don't know.

>
>> Alternatively, if there is a way of reliably identifying those
>> packages that can't be installed from source by someone without a
>> compiler, I'd be interested to know.
>
> Depends on how reliable you want it. Whatever mechanism someone can
> propose, I can find a way to cheat that mechanism.

I don't think anyone has a motive to cheat, unless their goal
if to be branded a cheater. :)  After all, as you point out,
if they cheat, they'll be found out pretty quickly.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Adding a "Pure Python" flag to PyPI

Donald Stufft
I'm +1 for this, and for packaging to do it automatically.

On Monday, March 5, 2012 at 4:43 PM, Jim Fulton wrote:

On Mon, Mar 5, 2012 at 4:21 PM, "Martin v. Löwis" <[hidden email]> wrote:
I can't see a way of reliably establishing whether a distribution is
"pure Python", and yet distutils/packaging clearly has that
information available when building. Would it be worthwhile adding a
"pure Python" flag to the PyPI classifiers, which could be
automatically populated by packaging? We'd still be reliant on people
who manually maintain metadata getting it correct, but it would help
in many cases (and in particular, in those cases where projects do
regularly upload binary distributions).

I don't think it's worthwhile. It would take forever (literally decades)
for this to get into wide use, unless some tool enforces it (e.g. PyPI
refuses the upload if there is a C file in the source tarball, yet the
package is not marked pure C).

distutils/2/packaging should be able to set this automatically.

If we do this, maybe the possible values should be yes, no, and don't know.


Alternatively, if there is a way of reliably identifying those
packages that can't be installed from source by someone without a
compiler, I'd be interested to know.

Depends on how reliable you want it. Whatever mechanism someone can
propose, I can find a way to cheat that mechanism.

I don't think anyone has a motive to cheat, unless their goal
if to be branded a cheater. :) After all, as you point out,
if they cheat, they'll be found out pretty quickly.

Jim

--
Jim Fulton
_______________________________________________
Distutils-SIG maillist - [hidden email]


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Adding a "Pure Python" flag to PyPI

Hanno Schlichting-4
In reply to this post by "Martin v. Löwis"
On Mon, Mar 5, 2012 at 10:21 PM, "Martin v. Löwis" <[hidden email]> wrote:
> However, downloading the source distribution and inspecting it should
> be fairly reliable.

Depends on what you want to find out. I know a bunch of packages which
contain optional C extensions. These certainly aren't pure-python and
contain C files in their source distributions. But during "setup.py
install" they'll catch any compilation errors and skip the extension
build step. So they are perfectly usable without a C compiler. Or they
employ even more logic in their setup.py files and make the C
extension conditional on the current platform - like disabling them in
PyPy or Jython.

Just two examples are zope.interface and Jinja2, which has an entirely
optional C extension via setuptools feature support.

I guess all I'm asking for is a better name that makes the intent clearer.

Hanno
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig