py2app and nested packages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

py2app and nested packages

Chris Barker
Folks,

py2app doesn't seem to do "the right thing" with nested packages.
However, it's not entirely clear what it should do...

example: the "pubsub" package is delivered with wxPython, so it is
commonly imported thusly:

from wx.lib.pubsub import Publisher

However, in wx.lib.pubsub, there is some trickery in the __init___, so
as py2app installs it, ot doesn't work (see previous messages to this
and the wxPython list). However, if you do:

packages = ['wx.lib.pubsub']  ...

in your py2app setup.py, you get the entire pubsub package, but installed in :

Peppy.app/Contents/Resources/lib/python2.7/pubsub

so it will work, but only if you now import it as:

from pubsub import Publisher

not too heinous, but it's really nice if we don't have to change
imports for the bundled version.

if I add 'wx' to packages, It does work, but I've then got all of wx,
all of wx.lib, etc, which is  really a bit much (the resulting app
bundle is 124MB, vs. 60 without specifying the wx package)

At first, I thought I'd want:

wx/lib/pubsub/....

put on sys.path, with wx and lib empty except the specified dirs.
However, that would put two "wx" dirs on sys.path, which would not be
good (assuming wx is used...)

Would it be possible for the packages specification to be processed
after modulegraph is run, and it would then see that "wx" and "wx.lib"
are there, and then put all of pubsub in there? Or maybe put the parts
of wx and wx.lib that were already going to be included outside the
zip bundle?

-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Chris Barker
Another couple py2app questions:

I"ve discovered that my app is HUGE! -- I already expected that, but
this is a bit out of bounds. The reason:

The scipy and matplotlib packages bring in all of scipy, numpy, and
matplotlib uncompressed. This gets pretty darn big, particularly when
you're using one or two simple functions from scipy, for instance.

I know why they do that -- there are a number of dynamic inports, and
folks often include matplotlib and/or scipy in apps where people may
be scripting, etc, and using features that aren't always imported by
the app. And this way, apps using those libs "just work" which is
great.

However, it would be nice to be able to override soem of this:

Is there a way to "turn off" a recipe (other than deleting it from the install)?

- Is there a way to apply excludes after the recipe? As far as I can
tell, if the recipe includes it, excludes doesn't remove it.


-Chris



On Mon, Apr 23, 2012 at 11:42 AM, Chris Barker <[hidden email]> wrote:

> Folks,
>
> py2app doesn't seem to do "the right thing" with nested packages.
> However, it's not entirely clear what it should do...
>
> example: the "pubsub" package is delivered with wxPython, so it is
> commonly imported thusly:
>
> from wx.lib.pubsub import Publisher
>
> However, in wx.lib.pubsub, there is some trickery in the __init___, so
> as py2app installs it, ot doesn't work (see previous messages to this
> and the wxPython list). However, if you do:
>
> packages = ['wx.lib.pubsub']  ...
>
> in your py2app setup.py, you get the entire pubsub package, but installed in :
>
> Peppy.app/Contents/Resources/lib/python2.7/pubsub
>
> so it will work, but only if you now import it as:
>
> from pubsub import Publisher
>
> not too heinous, but it's really nice if we don't have to change
> imports for the bundled version.
>
> if I add 'wx' to packages, It does work, but I've then got all of wx,
> all of wx.lib, etc, which is  really a bit much (the resulting app
> bundle is 124MB, vs. 60 without specifying the wx package)
>
> At first, I thought I'd want:
>
> wx/lib/pubsub/....
>
> put on sys.path, with wx and lib empty except the specified dirs.
> However, that would put two "wx" dirs on sys.path, which would not be
> good (assuming wx is used...)
>
> Would it be possible for the packages specification to be processed
> after modulegraph is run, and it would then see that "wx" and "wx.lib"
> are there, and then put all of pubsub in there? Or maybe put the parts
> of wx and wx.lib that were already going to be included outside the
> zip bundle?
>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> [hidden email]



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Ronald Oussoren
In reply to this post by Chris Barker

On 23 Apr, 2012, at 20:42, Chris Barker wrote:

> Folks,
>
> py2app doesn't seem to do "the right thing" with nested packages.
> However, it's not entirely clear what it should do...
>
> example: the "pubsub" package is delivered with wxPython, so it is
> commonly imported thusly:
>
> from wx.lib.pubsub import Publisher

wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems with py2app because it cannot detect this.  I added a recipe to py2app's repository a while ago that should fix this particular problem.

>
> However, in wx.lib.pubsub, there is some trickery in the __init___, so
> as py2app installs it, ot doesn't work (see previous messages to this
> and the wxPython list). However, if you do:
>
> packages = ['wx.lib.pubsub']  ...
>
> in your py2app setup.py, you get the entire pubsub package, but installed in :
>
> Peppy.app/Contents/Resources/lib/python2.7/pubsub
That's a bug, the code that implements the packages options assumes that all mentioned packages are toplevel packages.  It should be fairly easy to fix that, although this requires some path manipulation trickery because the subpackage will be in a different location that the parent: the parent is in site-packages.zip while the child is not.

>
> so it will work, but only if you now import it as:
>
> from pubsub import Publisher
>
> not too heinous, but it's really nice if we don't have to change
> imports for the bundled version.
>
> if I add 'wx' to packages, It does work, but I've then got all of wx,
> all of wx.lib, etc, which is  really a bit much (the resulting app
> bundle is 124MB, vs. 60 without specifying the wx package)
>
> At first, I thought I'd want:
>
> wx/lib/pubsub/....
>
> put on sys.path, with wx and lib empty except the specified dirs.
> However, that would put two "wx" dirs on sys.path, which would not be
> good (assuming wx is used...)
>
> Would it be possible for the packages specification to be processed
> after modulegraph is run, and it would then see that "wx" and "wx.lib"
> are there, and then put all of pubsub in there? Or maybe put the parts
> of wx and wx.lib that were already going to be included outside the
> zip bundle?
The option "packages" currently always stores the included package outside of site-packages.zip to ensure that data files can be loaded in the old pre-pkgresources way (that is, by opening paths relative to somepackage.__file__).

Ronald


_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Ronald Oussoren
In reply to this post by Chris Barker

On 23 Apr, 2012, at 23:13, Chris Barker wrote:

> Another couple py2app questions:
>
> I"ve discovered that my app is HUGE! -- I already expected that, but
> this is a bit out of bounds. The reason:
>
> The scipy and matplotlib packages bring in all of scipy, numpy, and
> matplotlib uncompressed. This gets pretty darn big, particularly when
> you're using one or two simple functions from scipy, for instance.
>
> I know why they do that -- there are a number of dynamic inports, and
> folks often include matplotlib and/or scipy in apps where people may
> be scripting, etc, and using features that aren't always imported by
> the app. And this way, apps using those libs "just work" which is
> great.
>
> However, it would be nice to be able to override soem of this:
>
> Is there a way to "turn off" a recipe (other than deleting it from the install)?
No, and I'd prefer to fix recipes instead of having a way to disable them.

The numpy/scipy recipes are probably as aggressive as they are because that fixes the immediate problem with minimal effort while writing the recipe.  The recipe should get a way to specify which dynamic imports are present, simular to the hardcoded knowlegde about the stdlib in modulegraph. That way py2app can include only the bits that are really necessary.

IIRC the matplotlib recipe includes the entire package because of the data files that are in the package, including only the code doesn't work.

>
> - Is there a way to apply excludes after the recipe? As far as I can
> tell, if the recipe includes it, excludes doesn't remove it.

That's correct, and is a bug: user specified actions should override automatic actions, including the recipes.

Ronald

>
>
> -Chris
>
>
>
> On Mon, Apr 23, 2012 at 11:42 AM, Chris Barker <[hidden email]> wrote:
>> Folks,
>>
>> py2app doesn't seem to do "the right thing" with nested packages.
>> However, it's not entirely clear what it should do...
>>
>> example: the "pubsub" package is delivered with wxPython, so it is
>> commonly imported thusly:
>>
>> from wx.lib.pubsub import Publisher
>>
>> However, in wx.lib.pubsub, there is some trickery in the __init___, so
>> as py2app installs it, ot doesn't work (see previous messages to this
>> and the wxPython list). However, if you do:
>>
>> packages = ['wx.lib.pubsub']  ...
>>
>> in your py2app setup.py, you get the entire pubsub package, but installed in :
>>
>> Peppy.app/Contents/Resources/lib/python2.7/pubsub
>>
>> so it will work, but only if you now import it as:
>>
>> from pubsub import Publisher
>>
>> not too heinous, but it's really nice if we don't have to change
>> imports for the bundled version.
>>
>> if I add 'wx' to packages, It does work, but I've then got all of wx,
>> all of wx.lib, etc, which is  really a bit much (the resulting app
>> bundle is 124MB, vs. 60 without specifying the wx package)
>>
>> At first, I thought I'd want:
>>
>> wx/lib/pubsub/....
>>
>> put on sys.path, with wx and lib empty except the specified dirs.
>> However, that would put two "wx" dirs on sys.path, which would not be
>> good (assuming wx is used...)
>>
>> Would it be possible for the packages specification to be processed
>> after modulegraph is run, and it would then see that "wx" and "wx.lib"
>> are there, and then put all of pubsub in there? Or maybe put the parts
>> of wx and wx.lib that were already going to be included outside the
>> zip bundle?
>>
>> -Chris
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> [hidden email]
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> [hidden email]
> _______________________________________________
> Pythonmac-SIG maillist  -  [hidden email]
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Chris Barker
In reply to this post by Ronald Oussoren
On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren

>> example: the "pubsub" package is delivered with wxPython, so it is
>> commonly imported thusly:
>>
>> from wx.lib.pubsub import Publisher
>
> wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems with py2app because it cannot detect this.  I added a recipe to py2app's repository a while ago that should fix this particular problem.


I thought I had the latest -- I'll go check that out.

Also, there is a bit of trick if you force it to import it's dyanamic
module, but if you've solved that one, great!


> That's a bug, the code that implements the packages options assumes that all mentioned packages are toplevel packages.  It should be fairly easy to fix that, although this requires some path manipulation trickery because the subpackage will be in a different location that the parent: the parent is in site-packages.zip while the child is not.

right -- do you have an idea how to solve that? I don't!

> The option "packages" currently always stores the included package outside of site-packages.zip to ensure that data files can be loaded in the old pre-pkgresources way (that is, by opening paths relative to somepackage.__file__).

which is useful, but another thought -- maybe there could be two options:

"packages" and "full_packages" or something -- so one would simple add
the package to the modulegraph, to catch dynamic imports, and the
other would do what is done now -- include the whole darn thing.

One other issue with the "whole darn thing" option, is that you get
the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
it could be -- I wonder if it's worth cleaning that up.

>> Is there a way to "turn off" a recipe (other than deleting it from the install)?

> No, and I'd prefer to fix recipes instead of having a way to disable them.

I understand that, and generally agree -- but it may be that there is
no one recipe that works for everyone. For example, Scipy is weird
enough that probably the only way to be sure it'll work in all cases
is to include the whole darn thing -- but in my case at least, I'm
using a couple functions in scipy.special, and it works fine if that
is all I include.

Maybe if there is a way to pass options to recipes?

> The (scipy) recipe should get a way to specify which dynamic imports are present, simular to the hardcoded knowledge about the stdlib in modulegraph. That way py2app can include only the bits that are really necessary.

If you give me a pointer - maybe to the similar modulegraph code,
perhaps I could work on that.

>> IIRC the matplotlib recipe includes the entire package because of the data files that are in the package, including only the code doesn't work.

I've poked into this more -- it turn out that there is special-case
code in matplotlib itself for py2exe that looks for the data-files in
a different place when sys.frozen -- I'm adding some code for py2app
as well -- if I get that working, I can submit a patch to MPL and the
recipe.

Is there a way to specify data files in a recipe?

>>- Is there a way to apply excludes after the recipe? As far as I can
>> tell, if the recipe includes it, excludes doesn't remove it.

> That's correct, and is a bug: user specified actions should override > automatic actions, including the recipes.

OK -- I suspect the issue is there is no way to exclude stuff that is
going to go outside of the zip file -- the full packages. It looks
like what it does is remove it from modulegraph only.

Thanks for your work (and ideas) on this.

-Chris




--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Ronald Oussoren

On 25 Apr, 2012, at 2:17, Chris Barker wrote:

> On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren
>
>>> example: the "pubsub" package is delivered with wxPython, so it is
>>> commonly imported thusly:
>>>
>>> from wx.lib.pubsub import Publisher
>>
>> wx.lib.pubsub is a bad example, it manipulates __path__ which causes problems with py2app because it cannot detect this.  I added a recipe to py2app's repository a while ago that should fix this particular problem.
>
>
> I thought I had the latest -- I'll go check that out.
Knowing me I probably forgot to check in the updated recipe :-(.  I also lost track on whether or not all changes are released on PyPI.

>
> Also, there is a bit of trick if you force it to import it's dyanamic
> module, but if you've solved that one, great!

IIRC that dynamic import is completely unnecessary and probably won't work when wx is in a zip file.

>
>
>> That's a bug, the code that implements the packages options assumes that all mentioned packages are toplevel packages.  It should be fairly easy to fix that, although this requires some path manipulation trickery because the subpackage will be in a different location that the parent: the parent is in site-packages.zip while the child is not.
>
> right -- do you have an idea how to solve that? I don't!

The copying part is easy, just create the right directory structure (packages = ["wx.pubsub"] -> create "wx/pubsub" in the application bundle). Ensuring that __import__ finds the subpackage is harder, I'll have to experiment to find a working solution that won't cause nausea in anyone looking at the code.

>
>> The option "packages" currently always stores the included package outside of site-packages.zip to ensure that data files can be loaded in the old pre-pkgresources way (that is, by opening paths relative to somepackage.__file__).
>
> which is useful, but another thought -- maybe there could be two options:
>
> "packages" and "full_packages" or something -- so one would simple add
> the package to the modulegraph, to catch dynamic imports, and the
> other would do what is done now -- include the whole darn thing.
>
> One other issue with the "whole darn thing" option, is that you get
> the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
> it could be -- I wonder if it's worth cleaning that up.
Adding "full_packages"  for the current behavior and giving "packages" a semantics that's simular to "modules" would be useful.

>
>>> Is there a way to "turn off" a recipe (other than deleting it from the install)?
>
>> No, and I'd prefer to fix recipes instead of having a way to disable them.
>
> I understand that, and generally agree -- but it may be that there is
> no one recipe that works for everyone. For example, Scipy is weird
> enough that probably the only way to be sure it'll work in all cases
> is to include the whole darn thing -- but in my case at least, I'm
> using a couple functions in scipy.special, and it works fine if that
> is all I include.
>
> Maybe if there is a way to pass options to recipes?
I guess you're right, there has to be a way to configure recipes and that includes disabling them completely.

Matplotlib is another example where this would be useful: it should be possible to specify that your app uses a particular backend without including the entire tree.

>
>> The (scipy) recipe should get a way to specify which dynamic imports are present, simular to the hardcoded knowledge about the stdlib in modulegraph. That way py2app can include only the bits that are really necessary.
>
> If you give me a pointer - maybe to the similar modulegraph code,
> perhaps I could work on that.

The modulegraph code I refer to is modulegraph.find_moduels.get_implies (<https://bitbucket.org/ronaldoussoren/modulegraph/src/tip/modulegraph/find_modules.py>).  The wx recipe in the py2app tree (<https://bitbucket.org/ronaldoussoren/py2app/src/tip/py2app/recipes/wx.py>) basicly does the same thing from the outside.

>
>>> IIRC the matplotlib recipe includes the entire package because of the data files that are in the package, including only the code doesn't work.
>
> I've poked into this more -- it turn out that there is special-case
> code in matplotlib itself for py2exe that looks for the data-files in
> a different place when sys.frozen -- I'm adding some code for py2app
> as well -- if I get that working, I can submit a patch to MPL and the
> recipe.
>
> Is there a way to specify data files in a recipe?
Good question, and it seems the anwer is no (looking at <https://bitbucket.org/ronaldoussoren/py2app/src/00b275dbdb71/py2app/build_app.py#cl-603>).  Adding an option for that would be easy enough though.

>
>>> - Is there a way to apply excludes after the recipe? As far as I can
>>> tell, if the recipe includes it, excludes doesn't remove it.
>
>> That's correct, and is a bug: user specified actions should override > automatic actions, including the recipes.
>
> OK -- I suspect the issue is there is no way to exclude stuff that is
> going to go outside of the zip file -- the full packages. It looks
> like what it does is remove it from modulegraph only.
>
> Thanks for your work (and ideas) on this.

Speaking of ideas....   I'm pretty sure that py2app is not compatible with packaging/distutils2, and I don't want to maintain a codebase that works with all flavor of distutils (the current distutils, setuptools, distribute, distutils2 and packaging).  

In the medium term it would be better to make py2app  a standalone application instead of a distutils extension, with clear integration points for the packaging tools (such hooks for fetching dependencies and building extensions).  That will have to wait for the next release of pyobjc though, that process is moving  forward way too slow even without spending time on a py2app redesign.

Ronald

>
> -Chris
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> [hidden email]
> _______________________________________________
> Pythonmac-SIG maillist  -  [hidden email]
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: py2app and nested packages

Russell Owen
In article <[hidden email]>,
 Ronald Oussoren <[hidden email]> wrote:

> On 25 Apr, 2012, at 2:17, Chris Barker wrote:
>
> > On Tue, Apr 24, 2012 at 4:29 AM, Ronald Oussoren
> >> The option "packages" currently always stores the included package outside
> >> of site-packages.zip to ensure that data files can be loaded in the old
> >> pre-pkgresources way (that is, by opening paths relative to
> >> somepackage.__file__).
> >
> > which is useful, but another thought -- maybe there could be two options:
> >
> > "packages" and "full_packages" or something -- so one would simple add
> > the package to the modulegraph, to catch dynamic imports, and the
> > other would do what is done now -- include the whole darn thing.
> >
> > One other issue with the "whole darn thing" option, is that you get
> > the *.py, *.pyc, Na *.pyo files -- so it's maybe three times as big as
> > it could be -- I wonder if it's worth cleaning that up.
>
> Adding "full_packages"  for the current behavior and giving "packages" a
> semantics that's simular to "modules" would be useful.

Please consider retaining the current meaning for the current name and
pick a new name for the new meaning (if you can figure one out that
makes sense). That would preserve backwards compatibility.

Is the inclusion of .pyc and .pyo files a significant problem? If they
are not already present then they will be regenerated next time the
application is run. Though if some .py files are never loaded then the
associated .pyc and .pyo files are wasted space.

-- Russell

_______________________________________________
Pythonmac-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG