RFC: Binary Distribution Format for distutils2/packaging

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
>> Please can we have a new format that only has a Python version in the
>> filename if it matters?
>
> isn't that supposed to be the source release ?

Yes, basically - at least as far as I understand.

> Why would someone create a binary release when
> it's pure Python ?

I wish I knew. But people do - mostly egg format files. But I think
this is partly because of the confusion between
egg-as-distribution-format vs egg-as-directly-usable-object that PJE
alludes to in his emails.

If there's a binary egg available, how do I know whether it's needed
without trying a source install and seeing if it works? And what about
where it does, but doesn't install C accelerators because I have no
compiler? The source works, but the binary is better. How would an
automated installation process know? Binary installers for pure-Python
projects exist, and cause confusion. (Example - IPython, see
http://archive.ipython.org/release/0.12/)

> or are you thinking about the auto-installable feature
> for windows here ?

No. I used to care about this, but no longer. I'm happy now to install
anything that *can* be installed from source directly, using pip (or
pysetup, once it works).

> I think there are two different uses cases:
>
> - install a program or a lib manually -- where an auto-installer makes
> sense.
> - have a library installed by a tool, where an auto-installer is not really
> needed.

I'm not sure I'd characterise the cases like that, I tend to think of
them as "wants to integrate with system standard policies" vs "wants
to use Python specific approaches". But it's the same 2 cases. I used
to be in the first camp, but I've switched and these days I'm firmly
in the second.

> I was thinking bdist_wininst and bdist_msi would stay around for people that
> want to create self-installable archives,
> and we would discuss a new format that is not self-installable.

The key thing for me, as a user, is having packages that I cannot
build myself available to me in a format that I can use. And
secondarily to have an easy user experience. The following are
possible installation scenarios:

1. pip/pysetup install PKG - transparent and simple, for source
distributions that are pure-Python or I can build (no external
dependencies and I have C installed)
2. pip/pysetup install PKG - new-format supplied by author (I assume
the install command will auto-download the new format)
3. download egg/wininst file
   run converter to create new-format
   pip/pysetup install new-format-file
   tidy up the temporary files
   - egg or wininst supplied by author, assuming converters exist and
are supplied with Python (add a "find the converter" step if they are
3rd party)
4. Install as system installer - MSI supplied by author, and I'm
installing in system Python.
5. Cry - MSI supplied by author and I want to install in a virtualenv
or similar. Basically, it's impossible (or at least, not worth the
effort).

To give me the best user experience, authors have to start supplying
the new format. The hard part is getting authors to take up a new
format (after all, they probably want to keep the old format as well,
for users who haven't switched to pysetup, so you've just doubled
their work). Alternatively, integrate the converters directly into
pip/pysetup, so that eggs and wininsts are *transparently* converted
on download, making case 3 look just like case 2. But at that point,
what's the argument for authors distributing the new format, when the
old ones work just as well for new-style users?

The technical issues aren't hard, it's the social issues that need
careful consideration.

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
If there's a binary egg available, how do I know whether it's needed
without trying a source install and seeing if it works?

The egg will have a platform string in its name in that case.  Otherwise, it'll be just package-version-pyX.X.egg.

(Actually, on reflection, I'm not sure I understand your question: "needed" relative to what?  to downloading the source version?)


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
2012/3/15 PJ Eby <[hidden email]>:

> On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
>>
>> If there's a binary egg available, how do I know whether it's needed
>> without trying a source install and seeing if it works?
>
>
> The egg will have a platform string in its name in that case.  Otherwise,
> it'll be just package-version-pyX.X.egg.
>
> (Actually, on reflection, I'm not sure I understand your question: "needed"
> relative to what?  to downloading the source version?)

Correct. In the context of "I'll use source if I can". I "need" a
binary if there are C files to compile (even for things like optional
speedup code).

I guess there's an implied requirement that I can tell what I "need"
just by inspection of the types of distribution available.because
obviously there's always the option of trying a source install and
seeing how it works.

In a wider sense, I'm thinking about how something like "pysetup
install" should work if binary formats are supported. If a package
supplies source, and a variety of eggs, but not wininst or new-format
package, then what should pysetup install (or the user, if
auto-conversion isn't implemented) choose? From what you say above,
the answer appears to be "the egg if its name has a platform string,
otherwise the source". That's basically what I was getting at.

Thanks,
Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
In reply to this post by Paul Moore
On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:

> On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
>>> Please can we have a new format that only has a Python version in the
>>> filename if it matters?
>>
>> isn't that supposed to be the source release ?
>
> Yes, basically - at least as far as I understand.
>
>> Why would someone create a binary release when
>> it's pure Python ?
>
> I wish I knew. But people do - mostly egg format files. But I think
> this is partly because of the confusion between
> egg-as-distribution-format vs egg-as-directly-usable-object that PJE
> alludes to in his emails.

I sometimes create platform-independent eggs to indicate a Python-version
dependency.  Until d2/p, there was no other way to indicate dependence
on a particular Python version.

Note that the terminology is confusing.  I think eggs are defined
to be a "binary" distribution format, so eggs containing
C extensions are referred to as "platform-specific".

You raise a good point about how to deal with optional extensions.
There's no meta data to indicate whether there are optional extensions
that might guide an automated installer.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Chris Barker
In reply to this post by Paul Moore
> On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:

>> Why would someone create a binary release when
>> it's pure Python ?

There are a lot of users (Windows and Mac anyway) that like a nice
point+click installer, and don't know (and shouldn't have to) whether
there is any compiled code in there. It's nice to support that.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Marius Gedminas-2
In reply to this post by Jim Fulton
On Thu, Mar 15, 2012 at 07:40:36AM -0400, Jim Fulton wrote:

> On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
> > On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
> >>> Please can we have a new format that only has a Python version in the
> >>> filename if it matters?
> >>
> >> isn't that supposed to be the source release ?
> >
> > Yes, basically - at least as far as I understand.
> >
> >> Why would someone create a binary release when
> >> it's pure Python ?
> >
> > I wish I knew. But people do - mostly egg format files. But I think
> > this is partly because of the confusion between
> > egg-as-distribution-format vs egg-as-directly-usable-object that PJE
> > alludes to in his emails.
>
> I sometimes create platform-independent eggs to indicate a Python-version
> dependency.  Until d2/p, there was no other way to indicate dependence
> on a particular Python version.
Except for Trove classifiers, of course:

    'Programming Language :: Python',
    'Programming Language :: Python :: 2',
    'Programming Language :: Python :: 2.4',
    'Programming Language :: Python :: 2.5',
    'Programming Language :: Python :: 2.6',
    'Programming Language :: Python :: 2.7',
    'Programming Language :: Python :: 3',
    'Programming Language :: Python :: 3.1',
    'Programming Language :: Python :: 3.2',

Or do I misunderstand your requirements?

Marius Gedminas
--
No proper program contains an indication which as an operator-applied
occurrence identifies an operator-defining occurrence which as an
indication-applied occurrence identifies an indication-defining occurrence
different from the one identified by the given indication as an
indication-applied occurrence.
                -- ALGOL 68 Report

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
On Thu, Mar 15, 2012 at 12:58 PM, Marius Gedminas <[hidden email]> wrote:

> On Thu, Mar 15, 2012 at 07:40:36AM -0400, Jim Fulton wrote:
>> On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
>> > On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
>> >>> Please can we have a new format that only has a Python version in the
>> >>> filename if it matters?
>> >>
>> >> isn't that supposed to be the source release ?
>> >
>> > Yes, basically - at least as far as I understand.
>> >
>> >> Why would someone create a binary release when
>> >> it's pure Python ?
>> >
>> > I wish I knew. But people do - mostly egg format files. But I think
>> > this is partly because of the confusion between
>> > egg-as-distribution-format vs egg-as-directly-usable-object that PJE
>> > alludes to in his emails.
>>
>> I sometimes create platform-independent eggs to indicate a Python-version
>> dependency.  Until d2/p, there was no other way to indicate dependence
>> on a particular Python version.
>
> Except for Trove classifiers, of course:
>
>    'Programming Language :: Python',
>    'Programming Language :: Python :: 2',
>    'Programming Language :: Python :: 2.4',
>    'Programming Language :: Python :: 2.5',
>    'Programming Language :: Python :: 2.6',
>    'Programming Language :: Python :: 2.7',
>    'Programming Language :: Python :: 3',
>    'Programming Language :: Python :: 3.1',
>    'Programming Language :: Python :: 3.2',
>
> Or do I misunderstand your requirements?

None of the tools use Trove classifiers to make decisions
about what to download afaik.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Marius Gedminas-2
On Thu, Mar 15, 2012 at 02:19:45PM -0400, Jim Fulton wrote:

> On Thu, Mar 15, 2012 at 12:58 PM, Marius Gedminas <[hidden email]> wrote:
> > On Thu, Mar 15, 2012 at 07:40:36AM -0400, Jim Fulton wrote:
> >> On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
> >> > On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
> >> >> Why would someone create a binary release when
> >> >> it's pure Python ?
> >> >
> >> > I wish I knew. But people do - mostly egg format files. But I think
> >> > this is partly because of the confusion between
> >> > egg-as-distribution-format vs egg-as-directly-usable-object that PJE
> >> > alludes to in his emails.
> >>
> >> I sometimes create platform-independent eggs to indicate a Python-version
> >> dependency.  Until d2/p, there was no other way to indicate dependence
> >> on a particular Python version.
> >
> > Except for Trove classifiers, of course:
> >
> >    'Programming Language :: Python',
> >    'Programming Language :: Python :: 2',
> >    'Programming Language :: Python :: 2.4',
> >    'Programming Language :: Python :: 2.5',
> >    'Programming Language :: Python :: 2.6',
> >    'Programming Language :: Python :: 2.7',
> >    'Programming Language :: Python :: 3',
> >    'Programming Language :: Python :: 3.1',
> >    'Programming Language :: Python :: 3.2',
> >
> > Or do I misunderstand your requirements?
>
> None of the tools use Trove classifiers to make decisions
> about what to download afaik.
Do they make decisions based on the presence of versioned .egg files?
I'd expect the tools to fall back to the sdist if they couldn't find an
.egg for a particular Python version.  If I'm right, then uploading
binary eggs doesn't help either -- unless you refrain from uploading the
sdist to PyPI?

Marius Gedminas
--
H.323 has much in common with other ITU-T standards - it features a complex
binary wire protocol, a nightmarish implementation, and a bulk that can be used
to fell medium-to-large predatory animals.
        -- Anthony Baxter

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
On Fri, Mar 16, 2012 at 1:53 PM, Marius Gedminas <[hidden email]> wrote:

> On Thu, Mar 15, 2012 at 02:19:45PM -0400, Jim Fulton wrote:
>> On Thu, Mar 15, 2012 at 12:58 PM, Marius Gedminas <[hidden email]> wrote:
>> > On Thu, Mar 15, 2012 at 07:40:36AM -0400, Jim Fulton wrote:
>> >> On Wed, Mar 14, 2012 at 7:29 PM, Paul Moore <[hidden email]> wrote:
>> >> > On 14 March 2012 19:04, Tarek Ziadé <[hidden email]> wrote:
>> >> >> Why would someone create a binary release when
>> >> >> it's pure Python ?
>> >> >
>> >> > I wish I knew. But people do - mostly egg format files. But I think
>> >> > this is partly because of the confusion between
>> >> > egg-as-distribution-format vs egg-as-directly-usable-object that PJE
>> >> > alludes to in his emails.
>> >>
>> >> I sometimes create platform-independent eggs to indicate a Python-version
>> >> dependency.  Until d2/p, there was no other way to indicate dependence
>> >> on a particular Python version.
>> >
>> > Except for Trove classifiers, of course:
>> >
>> >    'Programming Language :: Python',
>> >    'Programming Language :: Python :: 2',
>> >    'Programming Language :: Python :: 2.4',
>> >    'Programming Language :: Python :: 2.5',
>> >    'Programming Language :: Python :: 2.6',
>> >    'Programming Language :: Python :: 2.7',
>> >    'Programming Language :: Python :: 3',
>> >    'Programming Language :: Python :: 3.1',
>> >    'Programming Language :: Python :: 3.2',
>> >
>> > Or do I misunderstand your requirements?
>>
>> None of the tools use Trove classifiers to make decisions
>> about what to download afaik.
>
> Do they make decisions based on the presence of versioned .egg files?
> I'd expect the tools to fall back to the sdist if they couldn't find an
> .egg for a particular Python version.  If I'm right, then uploading
> binary eggs doesn't help either -- unless you refrain from uploading the
> sdist to PyPI?

That's exactly what I do.  (I don't do this very often.)

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Ronald Oussoren
In reply to this post by Chris Barker

On 14 Mar, 2012, at 18:42, Chris Barker wrote:

>
>
> Universal binaries (i.e. more than one architecture in one binary)
> have never been properly supported by binary eggs/setuptools. I think
> it may be as simple as the naming convention -- the binary would be
> named according to the machine it was built on (i.e. PPC) but when you
> tried to install in on another machine, setuptools would look for one
> called, i.e. "x86" and not find it. There may be some other issues,
> too, but in any case,  we need to make sure the naming convention
> handles the multi-architecture case as well.
Could you eleborate on that? Distutils supports fat binaries just fine, and AFAIK setuptools and distribute inherit that support. There were some problems in earlier python versions, especially in the Apple provided installs in OSX 10.4, but those issues have been resolved years ago.

Ronald
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

smime.p7s (6K) Download Attachment
12