RFC: Binary Distribution Format for distutils2/packaging

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
Sometimes, binary distributions are useful. This is especially true
for Windows and maybe Macs.  (They may also be useful for other
platforms in closed environments, where the shortcomings of
distutils platform definitions aren't such a problem.)

Distutils2/packaging (d2/p :) doesn't support the egg format.

The bdist format is problematic because:

- bdist names don't include Python versions and
  can't be reliably parsed to get project versions and
  platforms.

- The presence of bdists alongside of source and egg
  distributions breaks setuptools based tools, like
  easy_install, pip and buildout because they are
  mistaken for source distributions with wacky
  version numbers.

In discussing this at the PyCon packaging sprint, Tarek and i came up
with the following options:

1) setuptools eggs

   - Have to support legacy meta-data format

2) bdist

   - Need to add python version for:
     - compatibility info
     - also provides delimeter between version # and platform
   - Need to update setuptools/distribute to handle (or ignore) them.

3) New egg-like format "pbd"

   - Arrange suffix so ignored by setuptools/distribute
     - new-style meta data
     - would be a zip file

   - Essentially, .egg format with new-style meta data and different
     suffix.

Option 3) looks the best to us, so we propose:

- Introduce a new binary distribution format with a ".pbd" suffix
  and an egg-like structure.

  An example file name:

  ZODB3-3.10.0b1-py2.6-macosx-10.4-x86_64.pbd

- Deprecate bdist format.

  D2/p will not support generation or installation of bdist
  distributions.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Carl Meyer-4
On 03/13/2012 05:18 PM, Jim Fulton wrote:
[snip]

> 1) setuptools eggs
>
>    - Have to support legacy meta-data format
>
> 2) bdist
>
>    - Need to add python version for:
>      - compatibility info
>      - also provides delimeter between version # and platform
>    - Need to update setuptools/distribute to handle (or ignore) them.
>
> 3) New egg-like format "pbd"
>
>    - Arrange suffix so ignored by setuptools/distribute
>      - new-style meta data
>      - would be a zip file
>
>    - Essentially, .egg format with new-style meta data and different
>      suffix.
>
> Option 3) looks the best to us, so we propose:
>
> - Introduce a new binary distribution format with a ".pbd" suffix
>   and an egg-like structure.
>
>   An example file name:
>
>   ZODB3-3.10.0b1-py2.6-macosx-10.4-x86_64.pbd
>
> - Deprecate bdist format.
>
>   D2/p will not support generation or installation of bdist
>   distributions.
In terms of distribution format, this sounds great; I think a clean
break and new format that existing systems will just naturally ignore is
the way to go.

Your message doesn't address the on-disk installed format. I hope that
the installed format (at least the d2/p default) will be consistent with
PEP 376 (unzipped and "flat"), not an egg-style importable zip file
relying on its own dedicated entry in a pth file.

Carl


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Tarek Ziadé-2
On 3/13/12 7:50 PM, Carl Meyer wrote:
On 03/13/2012 05:18 PM, Jim Fulton wrote:
[snip]
1) setuptools eggs

   - Have to support legacy meta-data format

2) bdist

   - Need to add python version for:
     - compatibility info
     - also provides delimeter between version # and platform
   - Need to update setuptools/distribute to handle (or ignore) them.

3) New egg-like format "pbd"

   - Arrange suffix so ignored by setuptools/distribute
     - new-style meta data
     - would be a zip file

   - Essentially, .egg format with new-style meta data and different
     suffix.

Option 3) looks the best to us, so we propose:

- Introduce a new binary distribution format with a ".pbd" suffix
  and an egg-like structure.

  An example file name:

  ZODB3-3.10.0b1-py2.6-macosx-10.4-x86_64.pbd

- Deprecate bdist format.

  D2/p will not support generation or installation of bdist
  distributions.
In terms of distribution format, this sounds great; I think a clean
break and new format that existing systems will just naturally ignore is
the way to go.

Your message doesn't address the on-disk installed format. I hope that
the installed format (at least the d2/p default) will be consistent with
PEP 376 (unzipped and "flat"), not an egg-style importable zip file
relying on its own dedicated entry in a pth file.

yeah the installation format is PEP 376 for *all* distributions, including the old ones:
whatever gets installed is converted into PEP 376 if it's not a distutils2-based project

We're only talking here about a replacement for a binary *distribution*.

distutils' bdist is broken (the name of the file does not contain the py version...) while
setuptools' egg would be a decent choice, it adds more legacy burden.

I am excited we had that discussion with Jim, 3) seems the way forward for us.


Cheers
Tarek


Carl



_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
In reply to this post by Jim Fulton


On Tue, Mar 13, 2012 at 8:18 PM, Jim Fulton <[hidden email]> wrote:
3) New egg-like format "pbd"

  - Arrange suffix so ignored by setuptools/distribute
    - new-style meta data
    - would be a zip file

  - Essentially, .egg format with new-style meta data and different
    suffix.

FWIW, if I were designing the .egg format today with 20/20 hindsight, I would use 'package-version.egg-info' instead of 'EGG-INFO' as the metadata directory.  It would make discovery and converting to flat (pip/legacy-style) installations easier, as you could just unzip to the target directory.

Apart from that, I'm not sure what else you'll actually gain by introducing a new format.  IIUC, what new-style metadata egg files lack can be determined programmatically from their contents or the existing metadata files, so there seems to be little reason not to at least consume existing egg files.

Conversely, egg files support arbitrary metadata -- which means you can include whatever additional new-style data you want...  so there seems little reason not to simply add whatever extra metadata you want to them.

But perhaps there was some consideration or use case I'm not aware of?


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
In reply to this post by Carl Meyer-4
On 14 March 2012 02:50, Carl Meyer <[hidden email]> wrote:
> In terms of distribution format, this sounds great; I think a clean
> break and new format that existing systems will just naturally ignore is
> the way to go.

Agreed. I created a patch a while ago for a "bdist_simple" format,
which has languished in the tracker for a while. It's a very basic
patch, and the format is essentially just a stripped down version of
the wininst format (a zip file with distinguished placeholder
directories at the top level). I'm not sure I understand Jim's issues
with the wininst format, so I can't respond as to why this wouldn't
address them.

I'd started writing a PEP for bdist_simple with Éric Araujo but
stalled because I didn't have time. If someone else takes up the baton
with a binary distribution format, that's fine with me.

> Your message doesn't address the on-disk installed format. I hope that
> the installed format (at least the d2/p default) will be consistent with
> PEP 376 (unzipped and "flat"), not an egg-style importable zip file
> relying on its own dedicated entry in a pth file.

I agree that a flat on-disk format is essential. It's what pip and the
standard tools use. The .pth based egg format appears to only have
benefits if you want to manage multiple versions, and as far as I can
see that requirement is now better satisfied via virtualenv/pip.

One other point - it's actually relatively easy to write a tool to
introspect egg and wininst format files, and unpack them manually into
the on-disk layout needed. I have the bones of such a thing for pip
(needs completing before I can release it).

I would suggest that for backward compatibility and transition
purposes, the following things need to be considered:

- There should be converters from egg and wininst formats to the new
format. Not everyone will start producing the new format immediately,
and we should offer users a migration path where packagers haven't
moved over. These should be trivial to write.
- There should also be a converter from MSI format, but that's a beast
to write and I'd say not worth the effort.
- External tools like pip will need to handle the new format, ideally.
Or at least keep out of the way when pysetup install is used. I'm
assuming that pysetup install works in a virtualenv.

Is the intention to promote the new format for older Python versions
as well, via distutils2, or will this be a purely Python 3.3 option?
If the former, conversion and compatibility is crucial, if the latter,
then less so but we need to avoid causing problems for early adopter
users who rely on packagers who haven't moved yet ("Lobby the package
maintainer to change" is never a helpful answer...)

The key here is that the new format needs to be a unifying effort - we
already have 3 binary formats, if this new format merely adds a
fourth, then in my view it's a failure. It needs to be the "one
obvious way" if it's to work.

Paul.

PS A couple of relevant python-dev threads, for people who haven't seen them:

http://mail.python.org/pipermail/python-dev/2011-October/114285.html
http://mail.python.org/pipermail/python-dev/2011-October/113956.html
http://mail.python.org/pipermail/python-dev/2011-October/114243.html
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
In reply to this post by Tarek Ziadé-2
On 14 March 2012 02:56, Tarek Ziadé <[hidden email]> wrote:
> distutils' bdist is broken (the name of the file does not contain the py
> version...)

?? bdist_wininst filenames contain the python version when it matters
(i.e., when there are binaries in the file).

One thing I dislike about eggs is that the filename includes a Python
version for all eggs, even when the package is pure python. I know
eggs contain .pyc files (which are version specific) but IMO they
shouldn't. Having a python version in the filename where it's not
necessary increases the maintenance burden on packagers, who have to
generate the extra files, and for users, who have to work out whether
the package actually is version specific, or whether they can just
install from source.

Please can we have a new format that only has a Python version in the
filename if it matters?

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
In reply to this post by PJ Eby
On Tue, Mar 13, 2012 at 11:00 PM, PJ Eby <[hidden email]> wrote:
...
> I'm not sure what else you'll actually gain by introducing
> a new format.  IIUC, what new-style metadata egg files lack can be
> determined programmatically from their contents or the existing metadata
> files, so there seems to be little reason not to at least consume existing
> egg files.

I'm not sure whether you're referring to the distribution format, or
the meta-data format, or both.

I don't really have an opinion, myself, about the meta data format.
Other have decided it's going to be different. <shrug>

Having decided to use a different meta data format, it makes sense to
use a different
distribution format.  While d2/p might support eggs for a while (not
sure of that), we
wouldn't want new distributions to be in egg format, so we need an alternative.

(Personally, I don't have an issue with the egg format, but the decision was
made by others to move away from it.)

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
In reply to this post by Paul Moore
On Wed, Mar 14, 2012 at 6:52 AM, Paul Moore <[hidden email]> wrote:

> On 14 March 2012 02:50, Carl Meyer <[hidden email]> wrote:
>> In terms of distribution format, this sounds great; I think a clean
>> break and new format that existing systems will just naturally ignore is
>> the way to go.
>
> Agreed. I created a patch a while ago for a "bdist_simple" format,
> which has languished in the tracker for a while. It's a very basic
> patch, and the format is essentially just a stripped down version of
> the wininst format (a zip file with distinguished placeholder
> directories at the top level). I'm not sure I understand Jim's issues
> with the wininst format, so I can't respond as to why this wouldn't
> address them.

My main issue is that I think we need a standard binary format that
isn't system dependent.

I see the wininst format as sort of a two-edged sword for libraries.
On the one hand, it makes installing libraries natural for Windows
users, on the other, it encourages poor hygiene.

...

>> Your message doesn't address the on-disk installed format. I hope that
>> the installed format (at least the d2/p default) will be consistent with
>> PEP 376 (unzipped and "flat"), not an egg-style importable zip file
>> relying on its own dedicated entry in a pth file.
>
> I agree that a flat on-disk format is essential. It's what pip and the
> standard tools use.

Um....nevermind.

> The .pth based egg format appears to only have
> benefits if you want to manage multiple versions, and as far as I can
> see that requirement is now better satisfied via virtualenv/pip.

I don't see how virtualenv/pip address that.  I assume you mean
that if 2 applications/scripts require two different versions of a package,
they should use two separate virtualenvs.  That feels kinda heavy to me,
but to each their own. :)

FTR, buildout uses eggs to support multiple package versions. It doesn't
use .pth files, but bakes necessary paths into generated scripts.

Buildout on d2/p will continue creating egg-like structures,
both to support multiple versions and to support caching and
sharing installed packages.

> One other point - it's actually relatively easy to write a tool to
> introspect egg and wininst format files, and unpack them manually into
> the on-disk layout needed. I have the bones of such a thing for pip
> (needs completing before I can release it).
>
> I would suggest that for backward compatibility and transition
> purposes, the following things need to be considered:
>
> - There should be converters from egg and wininst formats to the new
> format. Not everyone will start producing the new format immediately,
> and we should offer users a migration path where packagers haven't
> moved over. These should be trivial to write.

+1

...

> - External tools like pip will need to handle the new format, ideally.
> Or at least keep out of the way when pysetup install is used. I'm
> assuming that pysetup install works in a virtualenv.

My hope is that pip will eventually use d2/p.

> Is the intention to promote the new format for older Python versions
> as well, via distutils2,

Yes

> or will this be a purely Python 3.3 option?

No.

> If the former, conversion and compatibility is crucial,

+1

...

> The key here is that the new format needs to be a unifying effort - we
> already have 3 binary formats, if this new format merely adds a
> fourth, then in my view it's a failure. It needs to be the "one
> obvious way" if it's to work.
>
> Paul.
>
> PS A couple of relevant python-dev threads, for people who haven't seen them:
>
> http://mail.python.org/pipermail/python-dev/2011-October/114285.html
> http://mail.python.org/pipermail/python-dev/2011-October/113956.html
> http://mail.python.org/pipermail/python-dev/2011-October/114243.html

Thanks for sharing these.

You raise an interesting point in the last link.

It's reasonable to argue that this is only a windows problem.

It could be argued that, on other platforms:

- Consumers of applications should get application installers,
  possibly with embedded copies of Python.

- Consumers of libraries are developers who should be able
  to install development tools.

I find the notion that we only need a binary format for Windows
unsatisfying and unsettling, but if everyone else feels that way,
I can live with it.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
On 14 March 2012 15:43, Jim Fulton <[hidden email]> wrote:
> My main issue is that I think we need a standard binary format that
> isn't system dependent.

I didn't intend bdist_simple to be platform-specific. It's based on
bdist_wininst certainly, but I stripped out all the platform-specific
bits (the embedding in the executable). The resulting format is simply
a zip file with separate directories for each of the sysconfig target
locations.

> I see the wininst format as sort of a two-edged sword for libraries.
> On the one hand, it makes installing libraries natural for Windows
> users, on the other, it encourages poor hygiene.

Not sure what you mean by "poor hygiene", but if you mean "installing
everything into the system Python" then I agree, and that's why I want
something like virtualenv to have support for some binary format (via
pip, packaging, or whatever).

> You raise an interesting point in the last link.
>
> It's reasonable to argue that this is only a windows problem.

I'm a Windows-only user, so I only have experience of that platform,
and it definitely is an issue there. When I stated on python-dev that
binary distributions seemed to be only an issue on Windows, a number
of Unix users (Nick Coghlan in particular, IIRC) argued that Unix
users would also benefit from binary packages. So I tried to design
something platform-neutral, although it needs a non-Windows user to
take a look and confirm whether or not there are platform dependencies
I missed.

> It could be argued that, on other platforms:
>
> - Consumers of applications should get application installers,
>  possibly with embedded copies of Python.

>From my experience, that happens more often on Windows than elsewhere
(py2exe/cx_Freeze). I didn't think Unix people did that.

> - Consumers of libraries are developers who should be able
>  to install development tools.

That was Nick's point, that sometimes even on Unix dependencies can be
hard to install or complex to build, so there is value in binary
distributions in those cases.

> I find the notion that we only need a binary format for Windows
> unsatisfying and unsettling, but if everyone else feels that way,
> I can live with it.

So do I - but as you say, I can live with it (as long as "Windows
only" doesn't end up translating into "not worth bothering about"...)

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
In reply to this post by Paul Moore
On Wed, Mar 14, 2012 at 8:21 AM, Paul Moore <[hidden email]> wrote:
One thing I dislike about eggs is that the filename includes a Python
version for all eggs, even when the package is pure python. I know
eggs contain .pyc files (which are version specific) but IMO they
shouldn't.

Your objection would make sense if .egg files were simply a format for transmitting files to be installed someplace -- in which case they could've omitted the .pyc files.

.egg files were originally designed as a plugin distribution format, which means they're supposed to be drop-in usable to an app's plugin directory.

Which brings up a few questions, actually.  Will these .pbd files be drop-in usable in the same way?  If not, then they're not really replacing eggs.  The same is true if they don't offer plugin discovery metadata like entry points or EggTranslations.
 
Having a python version in the filename where it's not
necessary increases the maintenance burden on packagers, who have to
generate the extra files,

This is a bit of a misconception, perhaps originating in the fact that setuptools itself was always distributed as a collection of version-specific eggs.  This was done solely to ease the bootstrapping of setuptools itself (to avoid recursively invoking the distutils while trying to build another package), and isn't really necessary for easy_install or other tools.  You only need to generate eggs if you are supporting a binary application of some kind.

Again, this raises the question: is .pbd a load-and-go format for distributing and *running* Python code, or is it just a way to bundle compiled extensions along with source code to simplify installation?  I worry that some folks in the conversation may be thinking one thing, and some folks the other.


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Carl Meyer-4

On 03/14/2012 09:33 AM, PJ Eby wrote:
> Again, this raises the question: is .pbd a load-and-go format for
> distributing and *running* Python code, or is it just a way to bundle
> compiled extensions along with source code to simplify installation?  I
> worry that some folks in the conversation may be thinking one thing, and
> some folks the other.

I think it was already clarified that it is the latter - purely a
distribution format, intended for unzipped PEP-376 style installation,
not drop-in use. Which means that no, it isn't a full replacement for
eggs (I don't think that was the goal).

Carl


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Jim Fulton
On Wed, Mar 14, 2012 at 12:38 PM, Carl Meyer <[hidden email]> wrote:

>
> On 03/14/2012 09:33 AM, PJ Eby wrote:
>> Again, this raises the question: is .pbd a load-and-go format for
>> distributing and *running* Python code, or is it just a way to bundle
>> compiled extensions along with source code to simplify installation?  I
>> worry that some folks in the conversation may be thinking one thing, and
>> some folks the other.
>
> I think it was already clarified that it is the latter - purely a
> distribution format, intended for unzipped PEP-376 style installation,
> not drop-in use. Which means that no, it isn't a full replacement for
> eggs (I don't think that was the goal).

Right.  It is only a replacement for use of eggs as a binary distribution
(not installation) format.

Jim


--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
In reply to this post by Jim Fulton


On Wed, Mar 14, 2012 at 11:17 AM, Jim Fulton <[hidden email]> wrote:
On Tue, Mar 13, 2012 at 11:00 PM, PJ Eby <[hidden email]> wrote:
...
> I'm not sure what else you'll actually gain by introducing
> a new format.  IIUC, what new-style metadata egg files lack can be
> determined programmatically from their contents or the existing metadata
> files, so there seems to be little reason not to at least consume existing
> egg files.

I'm not sure whether you're referring to the distribution format, or
the meta-data format, or both.

I don't really have an opinion, myself, about the meta data format.
Other have decided it's going to be different. <shrug>

Having decided to use a different meta data format, it makes sense to
use a different
distribution format.  While d2/p might support eggs for a while (not
sure of that), we
wouldn't want new distributions to be in egg format, so we need an alternative.

(Personally, I don't have an issue with the egg format, but the decision was
made by others to move away from it.)

In that case, I think it's really important to explicitly spell out the requirements and use cases for the new format; different people use .egg files differently, and the discussion here already reflects these different perceptions.

The packaging PEPs cover only replacements for EGG-INFO/PKG-INFO and EGG-INFO/requires.txt.  AFAIK, they don't include replacements for:

* entry_points.txt -- plugin discovery
* namespace_packages.txt -- namespace package management
* dependency_links.txt -- suggesting additional locations for finding dependencies
* native_libs.txt, eager_resources.txt, zip-safe/zip-not-safe -- support for in-zipfile access to resources and native libraries

Which of these use cases will be supported by the new format, and which will not?  Dropping any of them essentially means that some people are not going to have a migration path.   (PEP 382 or 402 will eventually clear up the namespace_packages bit, but are unlikely to be backported to 2.x.)

Also, is there an official discovery API for finding additional metadata (such as would be needed by EggTranslations' i18n system built on egg metadata)?

I assume that you're aware of most of the above, as Zope has historically made use of at least the first two items on the list above.  But a PEP for this new format really ought to spell out the use cases it's going to handle, or not, and why (or why not).


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
In reply to this post by Jim Fulton
On Wed, Mar 14, 2012 at 12:42 PM, Jim Fulton <[hidden email]> wrote:
Right.  It is only a replacement for use of eggs as a binary distribution
(not installation) format.

Or to put it another way, it won't be a replacement for eggs at all.  It'll be a replacement for bdist_wininst.


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Chris Barker
In reply to this post by Paul Moore
On Wed, Mar 14, 2012 at 9:17 AM, Paul Moore <[hidden email]> wrote:
>> It's reasonable to argue that this is only a windows problem.

no -- it's a Mac OS-X problem, too. Indeed, a harder one, due to:

A) The Mac platform now has 4! architectures: PPC, PPC64, x86,
intel64. Granted, PPC is almost dead, PPC64 never saw much use, and
even 32 bit x86 is on the way out. Never the less -- at least 32 and
64bit intel are going to be around for a while.

B) OS-X support fat binaries, and the pyton.org binaries have been
built that way for ages.

C) OS-X moves forward, fast -- it's gets pretty ticky to build
binaries that run on older versions than the one you are building on
(it can be done, to a point, but it's hard to get right)

>> - Consumers of applications should get application installers,
>>  possibly with embedded copies of Python.

sure -- but people developers need to build those...

> >From my experience, that happens more often on Windows than elsewhere
> (py2exe/cx_Freeze). I didn't think Unix people did that.

not as much, but Mac developers do.

>> - Consumers of libraries are developers who should be able
>>  to install development tools.

In theory, yes, but:

1) there are folks that want to do a little python that don't have any
experience or interest in the whole C building thing -- and to get the
compiler on the Mac, you need to register with Mac Developer
connection, then download a MASSIVE binary -- it's not a trivial lift.

2) as a stated above, building binaries of packages that you can
re-distribute to other systems (py2app) is tricky -- even more so when
you need compiled dependencies (libjpeg, libfreetype, that sort of
thing)

So the short version is -- binary packages are important on the Mac.

Which brings up another point:

Universal binaries (i.e. more than one architecture in one binary)
have never been properly supported by binary eggs/setuptools. I think
it may be as simple as the naming convention -- the binary would be
named according to the machine it was built on (i.e. PPC) but when you
tried to install in on another machine, setuptools would look for one
called, i.e. "x86" and not find it. There may be some other issues,
too, but in any case,  we need to make sure the naming convention
handles the multi-architecture case as well.

-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Paul Moore
In reply to this post by PJ Eby
On 14 March 2012 16:58, PJ Eby <[hidden email]> wrote:
> On Wed, Mar 14, 2012 at 12:42 PM, Jim Fulton <[hidden email]> wrote:
>>
>> Right.  It is only a replacement for use of eggs as a binary distribution
>> (not installation) format.
>
>
> Or to put it another way, it won't be a replacement for eggs at all.  It'll
> be a replacement for bdist_wininst.

That is certainly my understanding. Do we need a replacement for eggs?
It seems to me that the people who use them (not me) are happy with
them as they stand. On the other hand, packaging currently doesn't
have *any* equivalent of bdist_wininst (other than to use the legacy
version unchanged) - specifically, there's no binary format that
pysetup install will use - so there's a reasonably clear need there.

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
On Wed, Mar 14, 2012 at 1:45 PM, Paul Moore <[hidden email]> wrote:
On 14 March 2012 16:58, PJ Eby <[hidden email]> wrote:
> On Wed, Mar 14, 2012 at 12:42 PM, Jim Fulton <[hidden email]> wrote:
>>
>> Right.  It is only a replacement for use of eggs as a binary distribution
>> (not installation) format.
>
>
> Or to put it another way, it won't be a replacement for eggs at all.  It'll
> be a replacement for bdist_wininst.

That is certainly my understanding. Do we need a replacement for eggs?

Nope.  My only concern is that unless this is made really clear, people are likely to think this *is* intended to replace eggs, confusing eggs-in-general with eggs-as-improved-bdist_dumb/wininst.

Other than that, replacing bdist_dumb and bdist_wininst with something more sane has my +1.  (I just want it clearly labeled as such, since all the discussion about possibly using the egg format may confuse the unwary.)

If I understand correctly, all this proposal does is update bdist_dumb to use a saner naming convention, standardize on zip format, and flatten the inner directory layout.  A sort of "bdist_dumb2", if you will.


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

PJ Eby
In reply to this post by Chris Barker


On Wed, Mar 14, 2012 at 1:42 PM, Chris Barker <[hidden email]> wrote:
Universal binaries (i.e. more than one architecture in one binary)
have never been properly supported by binary eggs/setuptools. I think
it may be as simple as the naming convention -- the binary would be
named according to the machine it was built on (i.e. PPC) but when you
tried to install in on another machine, setuptools would look for one
called, i.e. "x86" and not find it. There may be some other issues,
too, but in any case,  we need to make sure the naming convention
handles the multi-architecture case as well.

FWIW, all the setuptools platform compatibility code for Mac OS was contributed by Bob Ippolito, and I don't remember his rationale for not supporting fat binaries exactly, although at the time (almost a decade ago) there were only two architectures and things in general were much simpler then.  ;-)

If you settle on a naming convention that works, patches to the pkg_resources functions 'compatible_platforms()', 'get_build_platform()', and 'get_supported_platform()' are welcome.  These are the only parts of setuptools that need to "understand" platform strings as anything other than an opaque identifier.

(For that matter, if anybody comes up with a sane platform naming convention for any *other* platforms, patches for those are welcome too!)

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Zvezdan Petkovic
In reply to this post by Chris Barker

On Mar 14, 2012, at 1:42 PM, Chris Barker wrote:

> 1) there are folks that want to do a little python that don't have any
> experience or interest in the whole C building thing -- and to get the
> compiler on the Mac, you need to register with Mac Developer
> connection, then download a MASSIVE binary -- it's not a trivial lift.

Not any more.
You can install XCode from Mac App store as an application now instead of an installation bundle/image.
Also, the membership in the Mac Developer connection is not required.
Only if you are going to sign apps for Mac or iOS you need the membership.  For a casual user who needs a C compiler and finds XCode on the Mac App store it's not required.

        Zvezdan

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Binary Distribution Format for distutils2/packaging

Chris Barker
On Wed, Mar 14, 2012 at 3:05 PM, Zvezdan Petkovic <[hidden email]> wrote:
>> 1) there are folks that want to do a little python that don't have any
>> experience or interest in the whole C building thing -- and to get the
>> compiler on the Mac, you need to register with Mac Developer
>> connection, then download a MASSIVE binary -- it's not a trivial lift.
>
> Not any more.
> You can install XCode from Mac App store as an application now instead of an installation bundle/image.

nicer -- though I imagine it's still a huge download.

> Also, the membership in the Mac Developer connection is not required.
> Only if you are going to sign apps for Mac or iOS you need the membership.  For a casual user who needs a C compiler and finds XCode on the Mac App
store it's not required.

nice to know.

However, the fact remains that there are folks that just want to write
some Python code -- Also, aside from the compiler, it's still can be a
pain to build stuff that requires dependencies (yes, I know about
homebrew, macports, fink -- the point still stands)

I think it's important to support that part of the community.

NOTE: on Windows, you can simply install the right version of Visual
Studio Express, and get building capability there, too -- but there is
still a real desire for binaries.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
12