Twisted plugin system and Python packaging

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Twisted plugin system and Python packaging

Tristan Seligmann-3
This conversation originally started in a bug report against pip[1],
but I'm moving it here since it seems this might be a better venue;
you may wish to read the bug log for interest, but I'll summarize the
issue from scratch anyway.

Twisted has a plugin system (twisted.plugin[2]) which is used by
Twisted itself, as well as other projects (such as Axiom[3] and
Dosage[4]) to allow for pluggable extensibility. The implementation
looks for modules in a plugins package and then looks in those modules
for objects providing (in the zope.interface sense) the IPlugin
interface. For the sake of simplifying this explanation, I will only
refer to twisted.plugins in the rest of this mail, but assume that
everything applies similarly to plugin packages in other projects (eg.
axiom.plugins and dosage.plugins).

twisted/plugins/__init__.py sets __path__ as follows:

from twisted.plugin import pluginPackagePaths
__path__.extend(pluginPackagePaths(__name__))

The implementation of pluginPackagePaths[5] loops through the
directories on sys.path, and includes DIR/twisted/plugins/ for every
DIR on sys.path, *except* when DIR/twisted/plugins/__init__.py exists.
This means that if you drop a project dir into PYTHONPATH while
developing, the plugins from your project will be picked up, even
though they're not installed into the copy of Twisted you're using;
you can also have a project shipping plugins installed in
~/.local/lib/pythonX.Y/site-packages even though you're using a
site-wide install of Twisted in /usr/lib/pythonX.Y/site-packages.
However, if you have multiple versions of Twisted on sys.path (because
you have a newer version installed locally overriding the site-wide
one, for example), plugins from a different version of Twisted won't
be accidentally picked up.

Given the way __path__ is set, if you have a project that ships
Twisted plugins, it would be possible to install it to
.../site-packages/MyProject/myproject/... +
.../site-packages/MyProject/twisted/plugins/myproject_plugin.py, and
then include this in a .../site-packages/MyProject.pth. However, the
usual way to install your project is to install it to
.../site-packages/myproject/... +
.../site-packages/twisted/plugins/myproject_plugin.py; in other words,
directly into the existing Twisted installation. If you include
"twisted.plugins" in your setup.py, then this works fine with
distutils "setup.py install" as well as "pip install"; setuptools
"setup.py install" will install everything into an egg, which will
also work due to the way __path__ is set. However, since "twisted"
ends up in top_files.txt in the egg-info, "pip uninstall" will blow
away your whole Twisted install when uninstalling a project shipping
Twisted plugins that was installed with "pip install". Ironically, if
the project was installed with setuptools "setup.py install", there is
no problem, since pip just removes the egg that was installed, and
nothing else.

In the pip bug report, Carl Meyer suggests using the setuptools
"namespace packages" feature; this works around the immediate problem
(pip uninstall doesn't blow away your Twisted install anymore), but
causes somer othe problems of its own. First of all, code in
__init__.py is not functional for a namespace package, so the __path__
functionality described above ceases to function. Additionally,
twisted.plugins cannot be declared as a namespace package without also
declaring twisted as a namespace package, which thus also affects
twisted/__init__.py; the code in this module exports an __version__
attribute (identifying the version of Twisted) as well as importing
twisted.python.compat to ensure that various backwards compatibility
monkeypatches are installed.

So, how should Twisted and Twisted-related projects be packaged in
order to avoid these issues? Please bear in mind that the current
plugin system in Twisted was first introduced around March 2005
(replacing the even older plugin system in use at the time, I
believe), thus there are quite a number of users relying on this code;
any changes would need to be backwards-compatible to avoid causing
problems for all of the existing projects and users relying on the
functionality.

[1] https://github.com/pypa/pip/issues/355

[2] http://twistedmatrix.com/documents/current/core/howto/plugin.html

[3] https://launchpad.net/divmod-axiom

[4] https://launchpad.net/dosage

[5] http://twistedmatrix.com/trac/browser/trunk/twisted/plugin.py#L225
--
mithrandi, i Ainil en-Balandor, a faer Ambar
_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Twisted plugin system and Python packaging

PJ Eby
At 04:20 PM 9/21/2011 +0200, Tristan Seligmann wrote:
>If you include "twisted.plugins" in your setup.py, then this works
>fine with distutils "setup.py install" as well as "pip install";
>setuptools "setup.py install" will install everything into an egg,
>which will also work due to the way __path__ is set. However, since
>"twisted" ends up in top_files.txt in the egg-info, "pip uninstall"
>will blow away your whole Twisted install when uninstalling a
>project shipping Twisted plugins that was installed with "pip install".

This really sounds like a bug in pip; top_level.txt is not a
replacement for a proper uninstall log.

>So, how should Twisted and Twisted-related projects be packaged in
>order to avoid these issues? Please bear in mind that the current
>plugin system in Twisted was first introduced around March 2005
>(replacing the even older plugin system in use at the time, I
>believe), thus there are quite a number of users relying on this
>code; any changes would need to be backwards-compatible to avoid
>causing problems for all of the existing projects and users relying
>on the functionality.

I think you've answered your own question here: there *isn't* any way
to package Twisted-related projects in a way that avoids the issue,
due to the bug in pip.  It's not Twisted's fault that pip takes shortcuts here.

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Twisted plugin system and Python packaging

Glyph Lefkowitz

On Sep 24, 2011, at 11:01 PM, P.J. Eby wrote:

This really sounds like a bug in pip; top_level.txt is not a replacement for a proper uninstall log.

Thanks.  It appears that Pip's maintainers are (the teensiest bit grudgingly) coming around to this perspective as well: https://github.com/pypa/pip/issues/355#issuecomment-2182951

Hopefully we can get a solution that pretty much works everywhere, soon.

-glyph

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Twisted plugin system and Python packaging

PJ Eby
At 11:32 PM 9/24/2011 -0400, Glyph Lefkowitz wrote:

>On Sep 24, 2011, at 11:01 PM, P.J. Eby wrote:
>
>>This really sounds like a bug in pip; top_level.txt is not a
>>replacement for a proper uninstall log.
>
>Thanks.  It appears that Pip's maintainers are (the teensiest bit
>grudgingly) coming around to this perspective as well:
><https://github.com/pypa/pip/issues/355#issuecomment-2182951>https://github.com/pypa/pip/issues/355#issuecomment-2182951

You may want to note that Twisted isn't the only distribution that
ships partial packages that aren't namespace packages.  The logilabs
packages are (or at any rate were) distributed this way, with a
top-level package in one distribution, and some modules (without an
__init__) were shipped in other distributions.  (I could be wrong,
but I think the mx.* stuff works this way, too.)

(These are all packages that are distutils-only, though, or have
their own extensions, IIRC.)

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Twisted plugin system and Python packaging

Glyph Lefkowitz

On Sep 25, 2011, at 1:44 AM, P.J. Eby wrote:

You may want to note that Twisted isn't the only distribution that ships partial packages that aren't namespace packages.  The logilabs packages are (or at any rate were) distributed this way, with a top-level package in one distribution, and some modules (without an __init__) were shipped in other distributions.  (I could be wrong, but I think the mx.* stuff works this way, too.)

Thanks for that data point!  It is really good to know, I'll use it to bludgeon some people with next time this issue comes up (on this list, or on some Debian bug, or whatever) ;-).

(These are all packages that are distutils-only, though, or have their own extensions, IIRC.)

You mean "extensions to distutils", or C extensions?  I think Twisted qualifies for both of those, and we definitely want to remain installable without setuptools, although we do use it for one or two things if it's available now.

-glyph

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig
Reply | Threaded
Open this post in threaded view
|

Re: Twisted plugin system and Python packaging

PJ Eby
At 08:52 PM 9/25/2011 -0400, Glyph Lefkowitz wrote:
>On Sep 25, 2011, at 1:44 AM, P.J. Eby wrote:
>>(These are all packages that are distutils-only, though, or have
>>their own extensions, IIRC.)
>
>You mean "extensions to distutils", or C extensions?

The former.  The mx.* stuff has a mx_setup or some such distutils
extension.  Not sure about the ll.* stuff.

_______________________________________________
Distutils-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/distutils-sig