python-dectest — Improved unittest.TestCase

I have released the first version of dectest for Python (Github), an improved version of unittest.TestCase. It is a drop-in replacement with two improvements:

  • decorators for tests, setup, and teardown methods. This is not only more explicit than the “magic” names of unittest, it also allows multiple setup and teardown methods per class.
  • patch() helper method that calls unittest.mock.patch(), but stops the patch during teardown.

Python asserts and json-get with type hints

I released new versions of python-asserts  (0.8.2) and python-json-get (1.1.0) that include type hints, so that typecheckers like mypy will now be able to type check calls to those libraries. json-get uses asserts for its tests and it’s now possible to type check it without --ignore-missing-imports.

In addition, json-get now has a json_get_default() function that returns a default value instead of raising an exception if a path is not found.

dbupgrade – A Database Migration Tool – published

Yesterday I published dbupgrade, a simple database migration tool, written in Python. It allows you to put database migration files (simple SQL files with a special header) into a directory and upgrade your database schema. While this is a new project, it was extracted from an internal library I’ve been using for years and should work fairly reliably for MySQL/MariaDB and Postgres databases.

dbupgrade is available at github and at pypi or via pip: pip install dbupgrade.

Chipcard Woes

One of my customers, a large dental practice, had a problem from time to time when reading Krankenversicherungskarten (German health insurance card, KVK). The insurance number got all garbled up, which led to problems when reading the patient’s insurance data into the health care application. After their sysadmin and I analysed the situation it seemed that the application reading the data from the card and writing it to a file in a custom format was to blame. This program (KVKTool) was included with the Cherry card reader and had been modified by another consultant to write the read data to a file rather than displaying it on the screen.

This C++ program was a huge stinking pile. All card reading was done in a method called CKVKToolDlg::Onreadcard. The name CKVKToolDlg hints at it: This is the class responsible for showing the dialog. (Yes, it had been left in the program by the previous consultant. Only the dlg.DoModal() call had been commented out.) The method was a gigantic 837 lines long and features all kinds of goodies such as lots of copied-and-pasted code that had been modified in some instance to fix bugs, but not in others.

After a long refactoring session (which turned into a rewrite), I was able to spot the problem: a wrong workaround to bugs in a sizable amount of KVKs. Normally, the data on KVKs is structured into fields in an arbitrary order. These fields consist of a one-byte tag to identify the field, a one-byte length marker, and then n bytes of data, where nis the length. All KVK fields are identified by tags between 0x80 and 0x92. But all KVK fields are wrapped by a super-field called the “Versichten-Template” (insurant template, VT) with tag 0x60. Thus, data on a typical (non-buggy) KVK card would start like this:

0x60 0x89 0x85 0x09 0x53 0x65 0x62 0x61 …
VT tag VT len First name tag First name length Seba …

Now, buggy cards seem to insert a stray 0x81 byte after the 0x60 tag:

0x60 0x81 0x89 0x85 0x09 0x53 0x65 0x62 0x61 …
VT tag ??? VT len First name tag First name length Seba …

This code in the KVKTool was supposed to handle this problem:

// skip tag 60 and length
int i=2;

// some cards seem to have a damaged tag 0x89 after 0x81
if (responsearray[1] == 0x81)
    i++;

(Snipped the huge indentation.)

responsearray is a buffer that contains the KVK data and i is an index into that buffer. This code works fine in most circumstances. It works around cards with the stray byte. But there is one situation where it utterly fails: On correct cards where the Versichten-Template is exactly 129 bytes long. In this case, the first tag is skipped entirely, which usually is the insurant’s number. Since it was not, junk (random memory content) was written to the file in place of the insurant’s number. Not good, but hopefully fixed now.

VirtualBox on i386 with amd64 Kernel

I have recently started to use an amd64 kernel on my i386 Debian unstable system. Unfortunately, VirtualBox OSE does not work with that setup. When I try to start a virtual machine, it fails with an oblique error message:

RTR3Init failed with rc=-1912 (rc=-1912)

The VirtualBox kernel modules do not fit to this version of VirtualBox. The installation of VirtualBox was apparently not successful. Executing

‘/etc/init.d/vboxdrv setup’

should fix that problem. Make sure that you don’t mix the OSE version and the PUEL version of VirtualBox.

Debian bug #456391 explains the problem. In that report Michael Meskes alludes to running VirtualBox in an amd64 chroot jail, so I tried this myself. It works flawlessly, once I got it setup. Here is what I did (as root):

robinson:~# mkdir /srv/amd64
robinson:~# cdebootstrap --arch amd64 sid /srv/amd64 http://ftp.debian.org/debian/
[...]
robinson:~# chroot /srv/amd64
robinson:/# apt-get update
[...]
robinson:/# apt-get upgrade
[...]
robinson:/# apt-get install virtualbox-ose # add more packages here if needed
[...]
robinson:/# adduser --uid 1000 --no-create-home --disabled-password --disabled-login srittau
[...]
robinson:/# 

These commands install the base system and create a user account. Now I created a script called /usr/local/bin/amd64.sh:

#!/bin/sh

CHROOT=/srv/amd64

if test ! -e $CHROOT/dev/.udev; then
    mount -t none /dev $CHROOT/dev/ -o bind
fi
if test `ls $CHROOT/proc | wc -l` = "0"; then
    mount -t proc none $CHROOT/proc
fi
if test `ls $CHROOT/sys | wc -l` = "0"; then
    mount -t sysfs none $CHROOT/sys
fi
if test `ls $CHROOT/home | wc -l` = "0"; then
    mount --bind /home $CHROOT/home
fi
chroot $CHROOT sh -c "su - srittau"

Running sudo amd64.sh will now enter the chroot environment as user srittau where I can start virtualbox normally.

setuptools breaks PYTHONPATH

setuptools and egg files are a great way to distribute Python packages and programs. But today I stumbled over a really braindead design decision: setuptools overrides Python’s standard module search path algorithm in a very inconvenient way. Normally, when Python looks for a module or package, it first looks in the current directory, then in any path specified in the PYTHONPATH environment variable, and finally in the default search path. setuptools changes the lookup order to the following:

  1. the current directory
  2. egg files specified in PYTHONPATH
  3. egg files in the default search path
  4. regular modules in PYTHONPATH
  5. regular modules in the default search path

This means that an egg file in the default search path overrides a non-egg module in PYTHONPATH. I wonder what the rationale for that decision is (and it seems to be a deliberate decision). PYTHONPATH is a means to override the default search path and is perfectly suited for testing, developing, and forcing to use a particular (e.g. a bundled) version of a package. Not anymore! Now, the only way to override an egg is to create your own egg. And this is non-trivial and often means an extra step during development.

Here is where it bites me: I have created a library of custom classes, that I use in several projects. I break API frequently, since it is developed in a just-in-time fashion: whenever I need some functionality, I implement it. Therefore I bundle the library (or parts of it) with most projects. This works well for projects that go into a separate directory, like web projects. Nevertheless I use the library for some other projects that are installed system-globally. Otherwise I would need some way to namespace it for the different projects. Not my idea of fun. Normally, this is not a problem, since I can just set PYTHONPATH for all projects that have a local copy, at least when developing. Due to the strange path handling of setuptools, this method breaks and I have to work around it. Great …

gnome-keyring with Python

The documentation on gnome-keyring I discovered helped me to access it successfully with Python. I’ve written a small module that fetches and stores a username and a password for some server.

Some notes:

  • The attributes are freeform, but there a some common attributes for network hosts. These are: “user”, “domain”, “server”, “object”, “protocol”, “authtype”, and “port”. Actually there is a convenience API for network hosts.
  • libgnome-keyring requires that the application is correctly set using g_set_application_name. Unfortunately the Python module gnomekeyring does not do that for us and pygtk does not provide a function to set the application name. Therefore you need to include the module gtk to make it work. (Filed as bug 460515.)Update: g_set_application_name was added to pygtk in the 2.13 development version.
  • Python’s find_network_password_sync function does not allow None to be passed as arguments. The C API allows NULL to be passed here. This means that this function is basically unusable in Python, since you never want to provide all values. (Filed as bug 460518.)

Documentation for libgnome-keyring

I was trying to find documentation for libgnome-keyring for little project I am writing, which accesses a password-protected web service. Unfortunately there is no real documentation for it. No API documentation (well, there are a total of two functions documented), no tutorial. Finally I found this document in GNOME’s Subversion repository. Better then nothing, but why isn’t this part of the API documentation?

Filed as bug 460475.

Update: It seems that libgnome-keyring has gained documentation a few weeks ago, so far only found in Subversion. Kudos to the maintainers!

Selection in GtkTextBuffer

I’ve recently played around with GtkTextBuffer. It’s a rather nice text editing widget (or rather widget part). Unfortunately it misses one functionality, which is also missing from GtkEditable derived widgets: A signal for selection changes. There are two workarounds:

  1. You can setup a notification on the “has-selection” property like this:
    buffer.connect("notify", on_buffer_notify)
    
    def on_buffer_notify(buffer, prop):
        if prop.name == "has-selection":
            if buffer.get_has_selection():
                ...
            else:
                ...
    

    This method only works if you are interested in whether there is a selection or not, not if you are interested in any selection changes. Also, this method works only for GtkTextBuffers, not for GtkEditables.

  2. Tomboy uses a timeout to check the selection status:
    namespace Tomboy
    {
    	public class NoteWindow : ForcedPresentWindow 
    	{
                    ...
    		public NoteWindow (Note note) : 
    			base (note.Title) 
    		{
                            ...
    			// Sensitize the Link toolbar button on text selection
    			mark_set_timeout = new InterruptableTimeout();
    			mark_set_timeout.Timeout  = UpdateLinkButtonSensitivity;
                            ...
                    }
    
                    ...
    
    		void UpdateLinkButtonSensitivity (object sender, EventArgs args)
    		{
    			link_button.Sensitive = (note.Buffer.Selection != null);
    		}
    
                    ...
            }
    }
    

    This method works, but seems a bit complicated, when a simple signal should do the trick.

Filed as bug #448261.