Specify missing install-requires dependencies to develop egg?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Specify missing install-requires dependencies to develop egg?

Matthew Brett
Hi,

We have been working on using buildout with a develop package
depending on numpy, and ran into trouble because 'numpy' is not
specified in the package 'install_requires'.

Specifically, we have something like this:

[buildout]
parts =
    nibabel-target
    mypy

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel

[mypy]
recipe = zc.recipe.egg:script
eggs =
    nibabel
interpreter = py

'nibabel' needs to import numpy to install, but does not specify
'numpy' in its setup.py install_requires, because pip install of numpy
is fragile, and we wanted to error early in general for an attempted
pip install of nibabel, if numpy was not present.

But of course the script above raises an error for the
'nibabel-target' part, because of missing numpy:

-------------
RuntimeError: Cannot import package "numpy" - is it installed?
While:
  Installing nibabel-target.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
------------

What we would very much like to do, is to be able to specify the
missing install_requires for this package (and for others we are
thinking of working with), in the recipe - something like:

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel
install-requires =
    numpy

but we couldn't immediately see a way of specifying such a dependency.
  A quick look at the Develop code in zc/recipe/egg/custom.py
suggested that there wasn't an easy way to add some needed egg onto
the path at this point.

Is there anything we missed that we should have tried?

If not, what would be the best way to solve our problem?  A modified recipe?

Best,

Matthew
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Reinout van Rees
On 26-08-11 03:06, Matthew Brett wrote:
> [mypy]
> recipe = zc.recipe.egg:script
> eggs =
>      nibabel
> interpreter = py
>
...
>
> What we would very much like to do, is to be able to specify the
> missing install_requires for this package (and for others we are
> thinking of working with)

The solution is quite simple: just add numpy to the list of eggs like this:

[mypy]
recipe = zc.recipe.egg:script
eggs =
     nibabel
     numpy
interpreter = py




Reinout

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Matthew Brett
Hi,

On Thu, Aug 25, 2011 at 7:06 PM, Reinout van Rees <[hidden email]> wrote:

> On 26-08-11 03:06, Matthew Brett wrote:
>>
>> [mypy]
>> recipe = zc.recipe.egg:script
>> eggs =
>>     nibabel
>> interpreter = py
>>
> ...
>>
>> What we would very much like to do, is to be able to specify the
>> missing install_requires for this package (and for others we are
>> thinking of working with)
>
> The solution is quite simple: just add numpy to the list of eggs like this:
>
> [mypy]
> recipe = zc.recipe.egg:script
> eggs =
>    nibabel
>    numpy
> interpreter = py

That doesn't work for me. Rightly or wrongly I am guessing that the
problem is that the dependency is not for the interpreter, but for
running nibabel setup.py.  In any case the failure remains with that
addition:

Updating nibabel-target.
Traceback (most recent call last):
  File "/tmp/tmpnKAmK6", line 11, in <module>
    execfile('/home/mb312/tmp/buildbotting/src/nibabel/setup.py')
  File "/home/mb312/tmp/buildbotting/src/nibabel/setup.py", line 40, in <module>
    package_check('numpy', NUMPY_MIN_VERSION)
  File "/home/mb312/tmp/buildbotting/src/nibabel/nisext/sexts.py",
line 122, in package_check
    raise RuntimeError(msgs['missing'] % pkg_name)
RuntimeError: Cannot import package "numpy" - is it installed?
While:
  Updating nibabel-target.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
...

Best,

Matthew
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Reinout van Rees
On 26-08-11 04:20, Matthew Brett wrote:

>>> What we would very much like to do, is to be able to specify the
>>> >>  missing install_requires for this package (and for others we are
>>> >>  thinking of working with)
>> >
>> >  The solution is quite simple: just add numpy to the list of eggs like this:
>> >
>> >  [mypy]
>> >  recipe = zc.recipe.egg:script
>> >  eggs =
>> >      nibabel
>> >      numpy
>> >  interpreter = py
> That doesn't work for me. Rightly or wrongly I am guessing that the
> problem is that the dependency is not for the interpreter, but for
> running nibabel setup.py.  In any case the failure remains with that
> addition:
>
> Updating nibabel-target.
> Traceback (most recent call last):
>    File "/tmp/tmpnKAmK6", line 11, in<module>
>      execfile('/home/mb312/tmp/buildbotting/src/nibabel/setup.py')
>    File "/home/mb312/tmp/buildbotting/src/nibabel/setup.py", line 40, in<module>
>      package_check('numpy', NUMPY_MIN_VERSION)
>    File "/home/mb312/tmp/buildbotting/src/nibabel/nisext/sexts.py",
> line 122, in package_check
>      raise RuntimeError(msgs['missing'] % pkg_name)
> RuntimeError: Cannot import package "numpy" - is it installed?
> While:
>    Updating nibabel-target.

"Updating nibabel-target": that's not the same part as that "mypy"
target. Note that such an eggs attribute is only valid for that one part.

So "[nibabel-target]" also should have an eggs attribute with numpy in it.

Perhaps handier: add an "eggs" attribute to [buildout] with nibabel and
numpy in it and use that:

[buildout]
....
eggs =
     nibabel
     numpy

[mypy]
...
eggs = ${buildout:eggs}
...

[nibabel]
...
eggs = ${buildout:eggs}
...



DOes that work?



Reinout

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Reinout van Rees
In reply to this post by Matthew Brett
On 26-08-11 03:06, Matthew Brett wrote:
> 'nibabel' needs to import numpy to install

With your latest reply I re-read your original. Hm, this looks like a
real problem: if your setup.py already imports from numpy (which your
statement sounds like), numpy really needs to be there already.

The only thing I can think of right now is to have numpy installed
globally and to use buildout 1.4.4 + osc.recipe.sysegg to depend on
numpy. If you call the sysegg part right at the beginning, you might get
it to work.


Alternative is to re-work your setup.py. Do you really need to import numpy?


Reinout

--
Reinout van Rees                    http://reinout.vanrees.org/
[hidden email]             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Matthew Brett
Hi,

On Thu, Aug 25, 2011 at 7:50 PM, Reinout van Rees <[hidden email]> wrote:
> On 26-08-11 03:06, Matthew Brett wrote:
>>
>> 'nibabel' needs to import numpy to install
>
> With your latest reply I re-read your original. Hm, this looks like a real
> problem: if your setup.py already imports from numpy (which your statement
> sounds like), numpy really needs to be there already.

Yes, that is what made me think that a new recipe might be the only
way, where the recipe would put numpy on the path during the develop
step.

> The only thing I can think of right now is to have numpy installed globally
> and to use buildout 1.4.4 + osc.recipe.sysegg to depend on numpy. If you
> call the sysegg part right at the beginning, you might get it to work.

Yes, if numpy is already on the path, it works, but we wanted to test
against several different numpy versions, and it seemed ugly and
fragile to make a separate virtualenv for each python / numpy version
pair.  We'd much rather use the buildout mechanism to specify these,
if possible.

> Alternative is to re-work your setup.py. Do you really need to import numpy?

Actually, no, not in this case, it is only used as a crude dependency
check.  But, for other packages we have, we do need numpy.distutils
for the setup.py - for example to find the numpy include directory for
compiling C modules.  The only reason I was offering this package as
an example is that it is rather small and easy to install compared to
the others.

Thanks a lot,

Matthew
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Jim Fulton
In reply to this post by Matthew Brett
On Thu, Aug 25, 2011 at 9:06 PM, Matthew Brett <[hidden email]> wrote:

> Hi,
>
> We have been working on using buildout with a develop package
> depending on numpy, and ran into trouble because 'numpy' is not
> specified in the package 'install_requires'.
>
> Specifically, we have something like this:
>
> [buildout]
> parts =
>    nibabel-target
>    mypy
>
> [nibabel-target]
> recipe = zc.recipe.egg:develop
> setup = src/nibabel
>
> [mypy]
> recipe = zc.recipe.egg:script
> eggs =
>    nibabel
> interpreter = py
>
> 'nibabel' needs to import numpy to install, but does not specify
> 'numpy' in its setup.py install_requires, because pip install of numpy
> is fragile, and we wanted to error early in general for an attempted
> pip install of nibabel, if numpy was not present.
>
> But of course the script above raises an error for the
> 'nibabel-target' part, because of missing numpy:
>
> -------------
> RuntimeError: Cannot import package "numpy" - is it installed?
> While:
>  Installing nibabel-target.
>
> An internal error occurred due to a bug in either zc.buildout or in a
> recipe being used:
> ------------
>
> What we would very much like to do, is to be able to specify the
> missing install_requires for this package (and for others we are
> thinking of working with), in the recipe - something like:
>
> [nibabel-target]
> recipe = zc.recipe.egg:develop
> setup = src/nibabel
> install-requires =
>    numpy
>
> but we couldn't immediately see a way of specifying such a dependency.
>  A quick look at the Develop code in zc/recipe/egg/custom.py
> suggested that there wasn't an easy way to add some needed egg onto
> the path at this point.
>
> Is there anything we missed that we should have tried?
>
> If not, what would be the best way to solve our problem?  A modified recipe?

Theoretically, you'd use setup_requires to specify packages needed by setup.
I'm not sure if that works with buildout. I've heard reports that it
does and reports that
it doesn't.

Why does nibabel want numpy at setup time? If it's to get C incluse files, then
you may be out of luck, as it's unlikely that buildout will install
numpy in a way that
will give nibabel what it wants.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Wichert Akkerman
On 08/26/2011 03:05 PM, Jim Fulton wrote:
> Theoretically, you'd use setup_requires to specify packages needed by setup.
> I'm not sure if that works with buildout. I've heard reports that it
> does and reports that
> it doesn't.

I'm pretty sure it doesn't. There is an open bug for it:
https://bugs.launchpad.net/zc.buildout/+bug/257555 . This bug looks
related: https://bugs.launchpad.net/zc.buildout/+bug/110133 .

Wichert.


_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Wichert Akkerman
On 08/26/2011 03:16 PM, Wichert Akkerman wrote:

> On 08/26/2011 03:05 PM, Jim Fulton wrote:
>> Theoretically, you'd use setup_requires to specify packages needed by
>> setup.
>> I'm not sure if that works with buildout. I've heard reports that it
>> does and reports that
>> it doesn't.
>
> I'm pretty sure it doesn't. There is an open bug for it:
> https://bugs.launchpad.net/zc.buildout/+bug/257555 . This bug looks
> related: https://bugs.launchpad.net/zc.buildout/+bug/110133 .

https://bugs.launchpad.net/zc.buildout/+bug/325658 is interesting as well.

Wichert.

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Specify missing install-requires dependencies to develop egg?

Matthew Brett
In reply to this post by Jim Fulton
Hi,

On Fri, Aug 26, 2011 at 6:05 AM, Jim Fulton <[hidden email]> wrote:

> On Thu, Aug 25, 2011 at 9:06 PM, Matthew Brett <[hidden email]> wrote:
>> Hi,
>>
>> We have been working on using buildout with a develop package
>> depending on numpy, and ran into trouble because 'numpy' is not
>> specified in the package 'install_requires'.
>>
>> Specifically, we have something like this:
>>
>> [buildout]
>> parts =
>>    nibabel-target
>>    mypy
>>
>> [nibabel-target]
>> recipe = zc.recipe.egg:develop
>> setup = src/nibabel
>>
>> [mypy]
>> recipe = zc.recipe.egg:script
>> eggs =
>>    nibabel
>> interpreter = py
>>
>> 'nibabel' needs to import numpy to install, but does not specify
>> 'numpy' in its setup.py install_requires, because pip install of numpy
>> is fragile, and we wanted to error early in general for an attempted
>> pip install of nibabel, if numpy was not present.
>>
>> But of course the script above raises an error for the
>> 'nibabel-target' part, because of missing numpy:
>>
>> -------------
>> RuntimeError: Cannot import package "numpy" - is it installed?
>> While:
>>  Installing nibabel-target.
>>
>> An internal error occurred due to a bug in either zc.buildout or in a
>> recipe being used:
>> ------------
>>
>> What we would very much like to do, is to be able to specify the
>> missing install_requires for this package (and for others we are
>> thinking of working with), in the recipe - something like:
>>
>> [nibabel-target]
>> recipe = zc.recipe.egg:develop
>> setup = src/nibabel
>> install-requires =
>>    numpy
>>
>> but we couldn't immediately see a way of specifying such a dependency.
>>  A quick look at the Develop code in zc/recipe/egg/custom.py
>> suggested that there wasn't an easy way to add some needed egg onto
>> the path at this point.
>>
>> Is there anything we missed that we should have tried?
>>
>> If not, what would be the best way to solve our problem?  A modified recipe?
>
> Theoretically, you'd use setup_requires to specify packages needed by setup.
> I'm not sure if that works with buildout. I've heard reports that it
> does and reports that
> it doesn't.

Ah - yes - thanks - I had forgotten about setup_requires.

> Why does nibabel want numpy at setup time? If it's to get C incluse files, then
> you may be out of luck, as it's unlikely that buildout will install
> numpy in a way that
> will give nibabel what it wants.

As I was saying in my previous message, nibabel just needs numpy at
runtime.  However, various of our other packages need it at setup
time, to find C include files, or to use numpy.distutils.

When running these various failing buildouts, numpy got installed as
an egg into the ./egg directory.  If I put that egg onto the path with
PYTHONPATH in my bare virtualenv, and then rerun this basic original
buildout.cfg

[buildout]
parts =
    nibabel-target
    mypy

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel

[mypy]
recipe = zc.recipe.egg:script
eggs =
    nibabel
interpreter = py

then it runs to completion.  With ./bin/py:

>>> import numpy
>>> numpy.get_include()
'/home/mb312/tmp/buildbotting/eggs/numpy-1.6.1-py2.6-linux-x86_64.egg/numpy/core/include'

- so I think the C includes would work.

Looking at the bugs pointed to by Wichert, it sounds like your
previous suggestion of a 'setup-requires' for custom and develop
recipes might be the way to go - that was what I was thinking of but
mis-naming in my previous mail.

Cheers,

Matthew
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig