python-dectest — Improved unittest.TestCase

I have released the first version of dectest for Python (Github), an improved version of unittest.TestCase. It is a drop-in replacement with two improvements:

  • decorators for tests, setup, and teardown methods. This is not only more explicit than the “magic” names of unittest, it also allows multiple setup and teardown methods per class.
  • patch() helper method that calls unittest.mock.patch(), but stops the patch during teardown.

Python asserts and json-get with type hints

I released new versions of python-asserts  (0.8.2) and python-json-get (1.1.0) that include type hints, so that typecheckers like mypy will now be able to type check calls to those libraries. json-get uses asserts for its tests and it’s now possible to type check it without --ignore-missing-imports.

In addition, json-get now has a json_get_default() function that returns a default value instead of raising an exception if a path is not found.

dbupgrade – A Database Migration Tool – published

Yesterday I published dbupgrade, a simple database migration tool, written in Python. It allows you to put database migration files (simple SQL files with a special header) into a directory and upgrade your database schema. While this is a new project, it was extracted from an internal library I’ve been using for years and should work fairly reliably for MySQL/MariaDB and Postgres databases.

dbupgrade is available at github and at pypi or via pip: pip install dbupgrade.

setuptools breaks PYTHONPATH

setuptools and egg files are a great way to distribute Python packages and programs. But today I stumbled over a really braindead design decision: setuptools overrides Python’s standard module search path algorithm in a very inconvenient way. Normally, when Python looks for a module or package, it first looks in the current directory, then in any path specified in the PYTHONPATH environment variable, and finally in the default search path. setuptools changes the lookup order to the following:

  1. the current directory
  2. egg files specified in PYTHONPATH
  3. egg files in the default search path
  4. regular modules in PYTHONPATH
  5. regular modules in the default search path

This means that an egg file in the default search path overrides a non-egg module in PYTHONPATH. I wonder what the rationale for that decision is (and it seems to be a deliberate decision). PYTHONPATH is a means to override the default search path and is perfectly suited for testing, developing, and forcing to use a particular (e.g. a bundled) version of a package. Not anymore! Now, the only way to override an egg is to create your own egg. And this is non-trivial and often means an extra step during development.

Here is where it bites me: I have created a library of custom classes, that I use in several projects. I break API frequently, since it is developed in a just-in-time fashion: whenever I need some functionality, I implement it. Therefore I bundle the library (or parts of it) with most projects. This works well for projects that go into a separate directory, like web projects. Nevertheless I use the library for some other projects that are installed system-globally. Otherwise I would need some way to namespace it for the different projects. Not my idea of fun. Normally, this is not a problem, since I can just set PYTHONPATH for all projects that have a local copy, at least when developing. Due to the strange path handling of setuptools, this method breaks and I have to work around it. Great …

gnome-keyring with Python

The documentation on gnome-keyring I discovered helped me to access it successfully with Python. I’ve written a small module that fetches and stores a username and a password for some server.

Some notes:

  • The attributes are freeform, but there a some common attributes for network hosts. These are: “user”, “domain”, “server”, “object”, “protocol”, “authtype”, and “port”. Actually there is a convenience API for network hosts.
  • libgnome-keyring requires that the application is correctly set using g_set_application_name. Unfortunately the Python module gnomekeyring does not do that for us and pygtk does not provide a function to set the application name. Therefore you need to include the module gtk to make it work. (Filed as bug 460515.)Update: g_set_application_name was added to pygtk in the 2.13 development version.
  • Python’s find_network_password_sync function does not allow None to be passed as arguments. The C API allows NULL to be passed here. This means that this function is basically unusable in Python, since you never want to provide all values. (Filed as bug 460518.)

New Python unittest module?

Collin Winter blogs about an updated unittest module he wrote. His update fixes the internal structure, and therefore the expandability, of the module, but also cleans up the external API. There are still a few minor improvements I would like to see, but nevertheless I hope that his updated version will be included in Python’s standard library eventually.

PyUnit and Decorators

One of the shortcomings of the Python standard library is in my opinion that PyUnit doesn’t use decorators (as later versions of JUnit and NUnit do). Therefore I have written a sample implementation that adds decorator support. (unittest.pydiff to Python 2.5’s unittest.py)

To do:

  • Warn when old-style and new-style tests are mixed.
  • Throw an exception when multiple setup or teardown methods exist.
  • Ability to warn when old-style methods are used.

XML Test Runner for PyUnit

JUnit features an XML Test Runner. This means that the result of a test run is not written to stdout, but instead written into an XML file. This XML file can then be automatically processed. This is for example useful for fully automated builds using software like CruiseControl. The XML output is easily converted into an XHTML file for easy human reading.

Until now PyUnit was lacking an XML Test Runner. Since I needed one for one of my projects that used CruiseControl, I wrote one. It is available for download here. Since this is an extension to a unit testing framework, the classes are of course fully tested. 🙂

If you have any suggestions or criticism, please let me know. I’m a nitpicker myself, so even if you have some small suggestions about coding style or improvement hints, I would like to hear about them.

Update: I have now submitted this patch to Python’s patch tracker, ticket number 1522704.

Update^2: I created a page with more information about PyUnit and CruiseControl integration, including sample configuration files and scripts.

Update^3: I updated the XmlTestRunner after input from Mirko Friedenhagen: It now recovers gracefully from unit tests overriding sys.stdout and sys.stderr, the XmlTestRunner returns the TestResult instance instead of a boolean value, and you can now stream multiple test suite results into a single XML file, since the XML header is not written to file streams by default.

Update 2017-10-30: While XMLTestRunner can be found on GitHub nowadays, its use has been deprecated for a while. Instead, use maintainer projects, such as unittest-xml-reporting.

Python Warts

I started my own page of Python warts, similar to A. M. Kuchling’s popular page. At the moment it just features one wart (the len() function vs. a length property on container classes). But I plan to add more warts in the future. I will probably concentrate on warts in the standard library. (And there are tons of them.) Two warts that come to mind are the unpythonic (but very SAXy) SAX library, and several warts in the unittest module.

Python Bindings for librsvg

Background: I started to re-write the old GNOME Chess application in Python. GNOME Chess is still a Gtk+ 1.2 application and as its Debian maintainer I want a modern version in Debian. Of course the chess board should be drawn with Cairo and the chess pieces were supposed to be SVG graphics. After a little research I noticed that the Debian package of pycairo did not enable SVG support. After I filed a bug report about this I found out that the cairo.svg module depends on the obsolete and not supported libsvg. Instead librsvg offers Cairo support and is the preferred solution. Only that there were no Python binding for librsvg.

So after fighting a whole day with pygtk, pyobject, and assorted tools, I am proud to present pyrsvg, Python bindings for librsvg. They are not really tested and probably need improvement. Please send any bugs or suggestions for improvement to me. In the long term I want this to become part gnome-python or gnome-python-extras.

Update: The bindings are now part of python-gnome-desktop (since version 2.15.1/2.16).