pip on windows: life without eggs ok?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

pip on windows: life without eggs ok?

Reinout van Rees
Hi,

I keep forgetting in discussions that pip doesn't support eggs. Their
main usecase is installing binary python eggs on windows, I'd say. I use
linux and OSX myself, so my windows experience is limited.

Question: is pip's compile-from-source character a problem on windows?
Does everyone use a clicky-click .exe installer for binary packages
instead of pip? Is easy_install still more popular?


Note, I'm not talking about installing pure python packages, but more
stuff like mapnik, pil, matplotlib and psycopg2: packages that have C
bindings.


I'm asking because I'm writing a Django book and I don't want to get
clobbered for advising pip when it won't install a mysql or postgres
binding without major work. So I'd love some feedback from windows users!


(I'm mostly using buildout myself, but I'm only describing that near the
end of my book).


Reinout

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Chris Lambacher-3
Hi,

If I were telling a new Windows Python user what to do to get started, I would tell them to use easy_install because easy_install will pick up both installers and binary eggs and do the right thing with those and virtualenvs.

pip on Windows is a decent into misery. Giving Windows users reliable instructions on how to set up an appropriate compiler is fraught with peril. Probably most new Python users on Windows have never done C or set up a build environment; they probably don't even know what a compiler is or why they would need one. Even if they do know about compilers, getting the right Visual Studio version(express or otherwise) is a bit of a problem because Python does not use the current version of VS and which version you need changes with Python versions. If you get a working compiler, then you need to track down the C dependencies of the module you are building.

The click installers are going to be a problem if you do any recommendation of virtualenv. The recent (or soon to be released) versions have --no-site-packages as the default and so, the click installers (that will install to global site-packages directory) won't show up without the user explicitly giving whatever the arg is for --with-site-packages when the virtualenv is created.

I heard rumours of plans for pip to support binary packages of some kind on Windows, but I don't know the details or current status of that.

-Chris

On Sun, Apr 15, 2012 at 7:50 PM, Reinout van Rees <[hidden email]> wrote:
Hi,

I keep forgetting in discussions that pip doesn't support eggs. Their main usecase is installing binary python eggs on windows, I'd say. I use linux and OSX myself, so my windows experience is limited.

Question: is pip's compile-from-source character a problem on windows? Does everyone use a clicky-click .exe installer for binary packages instead of pip? Is easy_install still more popular?


Note, I'm not talking about installing pure python packages, but more stuff like mapnik, pil, matplotlib and psycopg2: packages that have C bindings.


I'm asking because I'm writing a Django book and I don't want to get clobbered for advising pip when it won't install a mysql or postgres binding without major work. So I'd love some feedback from windows users!


(I'm mostly using buildout myself, but I'm only describing that near the end of my book).


Reinout

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig



--
Christopher Lambacher
[hidden email]

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Matthew Brett
Hi,

On Sun, Apr 15, 2012 at 8:07 PM, Chris Lambacher <[hidden email]> wrote:
> Hi,
>
> If I were telling a new Windows Python user what to do to get started, I
> would tell them to use easy_install because easy_install will pick up both
> installers and binary eggs and do the right thing with those and
> virtualenvs.
>
> pip on Windows is a decent into misery.

Seconded; our scientific packages use numpy (C) and scipy (C, C++
Fortran, big) and have C and sometimes Fortran extensions, so pip is
more or less useless for windows, and a real PITA for anything else.

Best,

Matthew
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Reinout van Rees
On 16-04-12 06:17, Matthew Brett wrote:

> Hi,
>
> On Sun, Apr 15, 2012 at 8:07 PM, Chris Lambacher<[hidden email]>  wrote:
>> Hi,
>>
>> If I were telling a new Windows Python user what to do to get started, I
>> would tell them to use easy_install because easy_install will pick up both
>> installers and binary eggs and do the right thing with those and
>> virtualenvs.
>>
>> pip on Windows is a decent into misery.
>
> Seconded; our scientific packages use numpy (C) and scipy (C, C++
> Fortran, big) and have C and sometimes Fortran extensions, so pip is
> more or less useless for windows, and a real PITA for anything else.

Thanks Chris and Matthew for the info! Especially about the upcoming
no-site-packages default.

I'm really going to have to add a section on debugging site.path to my
book... :-)


Reinout - happy to be using buildout ;-)

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Carl Meyer-4
Hi Reinout,

On 04/16/2012 04:31 AM, Reinout van Rees wrote:
> Thanks Chris and Matthew for the info! Especially about the upcoming
> no-site-packages default.

--no-site-packages has been the default since the release of virtualenv
1.7, which happened on Nov 30, 2011.

Carl


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Carl Meyer-4
In reply to this post by Chris Lambacher-3
Hi Chris,

On 04/15/2012 09:07 PM, Chris Lambacher wrote:
> If I were telling a new Windows Python user what to do to get started, I
> would tell them to use easy_install because easy_install will pick up
> both installers and binary eggs and do the right thing with those and
> virtualenvs.

Agreed.

> pip on Windows is a decent into misery. Giving Windows users reliable
> instructions on how to set up an appropriate compiler is fraught with
> peril. Probably most new Python users on Windows have never done C or
> set up a build environment; they probably don't even know what a
> compiler is or why they would need one. Even if they do know about
> compilers, getting the right Visual Studio version(express or otherwise)
> is a bit of a problem because Python does not use the current version of
> VS and which version you need changes with Python versions. If you get a
> working compiler, then you need to track down the C dependencies of the
> module you are building.
>
> The click installers are going to be a problem if you do any
> recommendation of virtualenv. The recent (or soon to be released)
> versions have --no-site-packages as the default and so, the click
> installers (that will install to global site-packages directory) won't
> show up without the user explicitly giving whatever the arg is for
> --with-site-packages when the virtualenv is created.
The argument is spelled --system-site-packages.

> I heard rumours of plans for pip to support binary packages of some kind
> on Windows, but I don't know the details or current status of that.

Support for installing binary packages on Windows has been a "patches
welcome" situation for as long as I've used pip (3-4 years?) - all
that's lacking is someone motivated to provide a good patch :-)

Carl


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Paul Moore
On 16 April 2012 20:08, Carl Meyer <[hidden email]> wrote:
>> I heard rumours of plans for pip to support binary packages of some kind
>> on Windows, but I don't know the details or current status of that.
>
> Support for installing binary packages on Windows has been a "patches
> welcome" situation for as long as I've used pip (3-4 years?) - all
> that's lacking is someone motivated to provide a good patch :-)

Speaking as someone who's looked at this, the problem seems to me to
be that pip has a *lot* of baked in assumptions that it's working with
source distributions, and it's very hard to see how to fit a binary
install mechanism in place. It would be (relatively) easier to add a
new "pip bininstall" subcommand, but once you start looking at that,
it's not easy to see what it gives you over "easy_install".

Personally, I'm waiting to see where the Python 3.3 "packaging"
solution fits into all this - as I understand it, it currently hits
the same spot as pip (source-install only) but it's less mature. It
may be that it will be easier to integrate binary installers into the
packaging infrastructure but there's still a lot of debate and no real
code in this area (I produced a patch, but it foundered over debates
about whether the binary format I proposed was "the right way" to do
it).

I'd love to see a group of people getting together to really work
through the options for binary installers that work with
pip/virtualenv/packaging, but the support seems pretty fragmented, and
the fact that eggs are a "good enough" solution tends to stall any
casual efforts in this direction.

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Carl Meyer-4
On 04/16/2012 03:56 PM, Paul Moore wrote:

> On 16 April 2012 20:08, Carl Meyer <[hidden email]> wrote:
>> Support for installing binary packages on Windows has been a "patches
>> welcome" situation for as long as I've used pip (3-4 years?) - all
>> that's lacking is someone motivated to provide a good patch :-)
>
> Speaking as someone who's looked at this, the problem seems to me to
> be that pip has a *lot* of baked in assumptions that it's working with
> source distributions, and it's very hard to see how to fit a binary
> install mechanism in place. It would be (relatively) easier to add a
> new "pip bininstall" subcommand, but once you start looking at that,
> it's not easy to see what it gives you over "easy_install".
I'm happy to consult with anyone working on this. I think this
particular framing conflates UI issues (what is the command named?) with
internal code structure unnecessarily. It's true that much of pip's code
is for working with sdists and support for installing binary packages
would require some significant new and separate code paths internally
(in InstallRequirement and, to a lesser extent, PackageFinder), but the
actual user-facing commands (e.g. InstallCommand) are a relatively small
part of the code; I don't think adding a separate command would actually
save much work over extending the "install" command. But I could be
wrong - were there specific issues that you saw as problematic for
adding support to "install"?

> Personally, I'm waiting to see where the Python 3.3 "packaging"
> solution fits into all this - as I understand it, it currently hits
> the same spot as pip (source-install only) but it's less mature. It
> may be that it will be easier to integrate binary installers into the
> packaging infrastructure but there's still a lot of debate and no real
> code in this area (I produced a patch, but it foundered over debates
> about whether the binary format I proposed was "the right way" to do
> it).
>
> I'd love to see a group of people getting together to really work
> through the options for binary installers that work with
> pip/virtualenv/packaging, but the support seems pretty fragmented, and
> the fact that eggs are a "good enough" solution tends to stall any
> casual efforts in this direction.
Agreed that focusing on support in d2/packaging is probably a better use
of time for the long term. I'm hopeful that at some point pip can lose a
lot of code and become a much thinner wrapper around d2/packaging, with
added support for things like installing direct from VCS, requirements
files, etc.

Carl


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pip on windows: life without eggs ok?

Paul Moore
On 16 April 2012 23:17, Carl Meyer <[hidden email]> wrote:

> I'm happy to consult with anyone working on this. I think this
> particular framing conflates UI issues (what is the command named?) with
> internal code structure unnecessarily. It's true that much of pip's code
> is for working with sdists and support for installing binary packages
> would require some significant new and separate code paths internally
> (in InstallRequirement and, to a lesser extent, PackageFinder), but the
> actual user-facing commands (e.g. InstallCommand) are a relatively small
> part of the code; I don't think adding a separate command would actually
> save much work over extending the "install" command. But I could be
> wrong - were there specific issues that you saw as problematic for
> adding support to "install"?

It was pretty much what you said - the InstallRequirement changes
would be fairly extensive to factor out the reusable parts from the
bits that only relate to source installs, and the finder changes
(which I hadn't located, but knew would need to be addressed) needed
to locate binary distributions as well as source ones. Also, pip
reuses setuptools to build source packages, but because easy_install
doesn't support --single-version-externally-managed, you can't do that
for a binary, so you have to replicate all that logic within pip.

Thanks, for the offer, though - if the packaging initiative doesn't
come up with anything concrete for 3.3, I might look at this again.
Although as I mentioned, I find that in practice, easy_install is good
enough for most cases where I need binaries, and hand-unpicking the
packages works for the rest (MSI and custom installers).

Paul.
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig