Python packaging: Where Pip and Virtualenv fall short

I love the combination of virtualenv and pip requirements files. This is a great 95% solution to the problem of deploying and distributing known-good sets of python packages, safely (semi-)isolated from other python environments on the same system.

The --no-site-packages option is great because it makes your installation isolated from everything except the globally-installed standard python library.

Here's the 5% where virtualenv falls painfully flat:

  • You can't always use --no-site-packages, or rather you can't always document that as the safest way for people to install your software, because you might have dependencies on C extensions (for me it's always lxml, gdal, and psycopg2), and --no-site-packages is tantamount to saying "you need a working C development toolchain and all the libraries and headers those C extensions are going to link against, and you better hope your system has working versions of all that, which sometimes it doesn't."

Using pre-built binary distributions of the C extensions generally removes a world of installation pain for people using your code, and that means you can't use --no-site-packages.

Now, I can now understand what's going on in those cases, and work around it. Unfortunately I am often distributing my software to people who are not python experts. Good luck explaining any of the above.

I wonder if zc.buildout has these or similar issues. I have no doubt that it gives the software distributor more power and more control, but it's also a hell of a lot more kool-aid to swallow, and I've never been able to bring myself to do it.

Meanwhile there are workarounds to the entry point / console script problems. If you specify a URL instead of a package version in a pip requirements file, it will get installed in your virtualenv. Checking out editable source works too. For example, if you depend on PasteScript 1.7.3:

# This will only get installed locally if it's *not* installed
# globally. Yikes.
pastescript==1.7.3

# This will definitely get installed.
http://pypi.python.org/packages/source/P/PasteScript/PasteScript-1.7.3.tar.gz

# So would this (note the @1.7.3 syntax to request a tag or branch)
-e hg+https://bitbucket.org/ianb/pastescript@1.7.3#egg=PasteScript-1.7.3

Caveat: I haven't done this in a long time but I vaguely recall that installing -e (or --editable) is different in at least one important way - it fails to install console scripts in the installed package.