setuptools and egg files are a great way to distribute Python packages and programs. But today I stumbled over a really braindead design decision: setuptools overrides Python’s standard module search path algorithm in a very inconvenient way. Normally, when Python looks for a module or package, it first looks in the current directory, then in any path specified in the PYTHONPATH environment variable, and finally in the default search path. setuptools changes the lookup order to the following:
- the current directory
- egg files specified in
PYTHONPATH
- egg files in the default search path
- regular modules in
PYTHONPATH
- regular modules in the default search path
This means that an egg file in the default search path overrides a non-egg module in PYTHONPATH. I wonder what the rationale for that decision is (and it seems to be a deliberate decision). PYTHONPATH is a means to override the default search path and is perfectly suited for testing, developing, and forcing to use a particular (e.g. a bundled) version of a package. Not anymore! Now, the only way to override an egg is to create your own egg. And this is non-trivial and often means an extra step during development.
Here is where it bites me: I have created a library of custom classes, that I use in several projects. I break API frequently, since it is developed in a just-in-time fashion: whenever I need some functionality, I implement it. Therefore I bundle the library (or parts of it) with most projects. This works well for projects that go into a separate directory, like web projects. Nevertheless I use the library for some other projects that are installed system-globally. Otherwise I would need some way to namespace it for the different projects. Not my idea of fun. Normally, this is not a problem, since I can just set PYTHONPATH for all projects that have a local copy, at least when developing. Due to the strange path handling of setuptools, this method breaks and I have to work around it. Great …