Unix and non-ASCII file names, a summary of issues

Unix/Linux file and directory names are binary byte C strings. Only the null byte and the slash character (/) are forbidden inside a name, nowhere does the kernel interpret the strings as meaningful or printable.

In the old times, all utilities that would display to the user were ASCII-based, and people would use pure printable ASCII file names (even using space characters inside names was a cause for trouble). Non alphanumeric characters were exclusively used for playing tricks on colleagues. And all was well.

Then the devil came under the guise of accented 8 bit characters. The system has no problem with them, file names are still binary C strings, but the utilities have to display them or take them as input, and, because there is no encoding specification stored with the file names, they can only do this according to the character encoding taken from the user’s current locale.

For example fr_FR.UTF-8, and fr_FR.ISO8859-1 could be used simultaneously on the same system (by different users), but they are completely uncompatible: ISO-8859-1 strings are illegal when viewed in an UTF-8 locale (will display as interrogation points or some other conventional error marker). UTF-8 strings will display as gibberish in an ISO-8859-1 locale.

This means that the file names created by an UTF-8 user are displayed as garbage to the ISO-8859 one…​

If you ever change your locale, your old files are still there and named the same (in the binary sense), but the names display badly and you have great trouble inputing them. If you add distributed (NFS) file system issues, things become totally unmanageable. Also think about archives sent from another system with a different encoding.

For what concerns Recoll:

  • The file names inside recoll.conf are not transcoded, they are taken as binary strings (mostly, only \n and space are a bit special), and passed as is to the system. So if you edit 'recoll.conf' with a text editor, inside the same locale that is or has been used for file names, you’ll be fine.

  • There was a bug in the GUI configuration tool, up to 1.12, it should transcode between the internal Qt format and locale-dependant strings, but it doesn’t or does it badly.

  • There is also an exception for the unac_except_trans variable, this has to be UTF-8, so if the rest of the file uses another encoding, you’ll need to edit two separate files and concatenate them.

As of version 1.13, Recoll uses local8Bit()/fromLocal8Bit() to convert recoll.conf file names from/to QStrings (it uses UTF-8 for all string values which are not file names).

The Qt file dialog is broken (at least was, I have not checked this on recent versions). It should consider file paths as almost-binary data, not QStrings, but doesn’t. In consequence, things are even more broken than necessary as seen from there:

With LANG="C", no non-ASCII paths can’t be used at all:

  • Strings read from recoll.conf are stripped of 8bit characters before display.

  • Directory entries with 8bit characters are not displayed at all in the selection dialog.

With LANG="fr_FR.UTF-8", only UTF-8 paths can be used:

  • Strings read from recoll.conf are damaged when converted to QString (except those that were actually UTF-8)

  • Only the UTF-8 directory entries are displayed in the selection dialog.

With LANG="fr_FR.iso8859-1", everything works ok.

  • Strings read from recoll.conf are displayed with weird characters if they use another encoding such as UTF-8, but are correctly maintained and can be read back from the dialogs and rewritten without damage.

  • Directory entries with 8 bit characters are displayed weirdly (normal), but can be manipulated without trouble (this includes utf-8 names of course).

In conclusion, only the iso-8859 locales can be used for handling mixed encoding situations. This is a possible workaround for people who need it.

More data about path encoding issues: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html