Ever since upgrading to a recent Mac that came with the disk formatted with AFPS, a perennial irritation has been Time Machine. I use a USB hard drive for backups, which of course needs unplugging when I want to take the machine with me somewhere. There are long stretches of time when I don’t even think about this because it works just fine. And then there are the other stretches of time when this has been impossible: clicking the eject button in Finder does nothing for a few ponderous moments and then shows a force eject dialog. (And of course the command line tools and other methods all equally fail.)

I could of course forcibly eject the disk, as the dialog offers. And maybe I would, if it this was just a USB stick I was using to shuffle around some files. But doing this with my backup disk seems rather less smart a move. This disk I want to unmount in good order.

Unfortunately when this happens, there is no help for it: even closing all applications does not stop the mystery program from using it. So what is the program which is using the disk? The Spotlight indexer, it turns out.

$ sudo lsof +d /Volumes/TimeMachine\ XXXXXX/
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mds     1234 root    5r   DIR   1,24      160    2 /Volumes/TimeMachine XXXXXX
mds     1234 root    6r   DIR   1,24      160    2 /Volumes/TimeMachine XXXXXX
mds     1234 root    7r   DIR   1,24      160    2 /Volumes/TimeMachine XXXXXX

How do you ask this to stop?

Beats me. I have not found any documented, official way of doing so. Not the Spotlight privacy settings, not removing the disk from the list of backup disks in the Time Machine settings, not the combination of those, no conceivable variation of using tmutil on the command line, not a number of other things – nothing. Even killall mds does not help: obviously the Spotlight service notices, and just respawns the processes.

And this state will persist for hours and days – literally. On one occasion, I wanted but didn’t need the machine with me, so I left it to its own devices out of curiosity. It took over 2 days before ejecting the Time Machine volume worked again.

For a purportedly portable computer, this is… you know… a bit of a showstopper.

So after suffering this issue long enough, I finally tried something stupid the other day – and whaddayaknow, it works:

$ sudo killall mds ; sudo umount /Volumes/TimeMachine\ XXXXXX/

This will not always work the first time, it may need a repeat or two. But sooner rather than later it does take. Evidently the mds process respawn is not so quick that it wouldn’t leave a window during which the disk can be unmounted properly.

And so I put the following in ~/bin/macos/unmount-despite-mds and made it executable:

#!/bin/bash
if (( $# != 1 )) ; then echo "usage: $0 <mountpoint>" 1>&2 ; exit 1 ; fi
parent_dev=$( stat -f %d "$1"/.. ) || exit $?
while [[ -d $1 ]] && [[ "$( stat -qf %d "$1" )" != $parent_dev ]] && ! umount "$1" ; do
  killall mds
  lsof +d "$1"
done

Now I can invoke it at any time from the terminal like so:

$ sudo ~/bin/macos/unmount-despite-mds /Volumes/TimeMachine\ XXXXXX/

What this does is check whether the given path, if it is a mount point, fails to unmount. If so, it sends a signal to terminate the Spotlight indexer processes and immediately retries. In a loop.

Or to put it more colloquially, it machineguns Spotlight until umount can slip in under the suppressing fire and pull the disk out from under it.

This is not a solution. It is the bluntest of instruments. But this is what works. And as far as I have been able to find, this is the only thing that works.

Three cheers for Apple software quality, I guess.

Some time ago I had a pile of tarballs created periodically from a cron job which were eating up the space on that machine. I needed a quick way to identify which of them had changes relative to the respective previous one.

I was hoping I could just compare the tarballs themselves rather than doing anything more complicated that involved actually extracting their contents (and then, I don’t know, doing some kind of fingerprinting on top). Unfortunately some naïve attempts using cmp failed and seemed to indicate that I was going to have to do something complicated like that.

Now there is quite a bit of discussion online about how to make Tar generate reproducible (so in a sense, canonical) tarballs. Of course that isn’t much use in hindsight (such as in my case), when a pile of tarballs is already sitting around on disk.

But in my case, the source tree was a part of the filesystem where absolutely nothing had happened. And I don’t just mean logical non-changes like creating and then deleting a temporary file. I mean that nothing was writing to that part of the filesystem in any capacity. Therefore Tar should encounter files and directories in the exact same order each time it was iterating that tree. Why then should it be generating non-identical tarballs?

Exasperated, I dug into the question of archive reproducibility for quite a while.

Spoiler: that was a waste of time. I finally discovered that Tar is not the culprit at all – Gzip is! Namely, the Gzip file header includes a timestamp. My instincts were right: Tar should have been creating the exact same archive over and over – and it was. I just hadn’t thought to suspect Gzip at all.

Luckily, the timestamp is found at a fixed location in a gzipped file: it is the 32-bit value at offset 4. And handily, cmp has a switch to tell it to seek past the start of the file(s) it is comparing. To make a long story short:

cmp -i8 file1.tar.gz file2.tar.gz

(This entry is brought to you by the hope to not have to figure this all out a third time in my life. Some time after the events described above, it came back to me that I had already figured this out years before but lost all memory of it by the next occasion to use the knowledge.)

Over 6 years have gone by since I cut the last release of rename because no new features have been needed in the time since. Unfortunately a number of small bugfixes and documentation additions have therefore sat unreleased in the repository for years. You have a bug report about a long-fixed issue to thank for me finally noticing. I am hereby rectifying this situation:

Share and enjoy.

Over on his weblog, Chris Siebenmann talks about metadata in file based websites:

In a file based website engine, any form of metadata that you can’t usefully commit into common version control systems is a mistake.

This is part of a class of mistake I’ve variously made or run into in a number of contexts.

A prominent example that comes to mind is an application I work on which tracks records of some kind, and in one place, users enter “number of X for the thing recorded here” as data points over time. Originally the point in time used for a data point was the time of entry of the data point. At some point I realised that this was a mistake: the point in time at which the number of X was Y is not the same as the point in time at which that fact was recorded in our system. Among other things, treating them the same means you cannot enter “number of X” data points after the fact (such as when newly capturing a record retroactively). One is metadata about a real-world event, the other is metadata about the usage of our application to record that real-world event.

So the generalised principle goes maybe something like this:

Metadata about a storage entity should not be confused with metadata about whatever the payload of that storage entity represents.

These may (obviously) coincide but are nevertheless not the same thing. If an article is stored in a file, the creation date of the file just says when that article was saved into that file – not when the article was written. As long as the “writing an article” and “saving it into a file” actions are strongly bound to each other, these metadata will coincide so that it may seem attractive to treat one as the other. But once you introduce ways for them to diverge – such as adding a VCS to the mix, which creates/removes/modifies files without a corresponding article-writing/editing action – you have a problem.