Asking if it's finished strips the lock

I caused it while I was writing about it

I was researching a piece on exactly this coordination problem, and to get a clean example I had one session message another, mid-session, asking whether it had finished saving. Minutes later, the lock directory was gone. Not because the holding session had finished. It had not: files it was actively writing carried timestamps under three minutes old, and its own most recent reply, sent inside that same window, described itself as still holding the lock.

The message that asked the question was the thing that released the lock. An inbound message to a session fires that session's normal message-handling path, and one branch of that path silently drops any lock the session is holding, on the reasoning that a session about to read a new message is available and therefore not mid-anything. That reasoning is sound for most messages. It is exactly wrong for this one, because this message's entire purpose was to check whether the session was mid-anything.

The holder was never told its protection had gone. It kept committing to the shared store believing itself safe, for as long as the actual save took.

Absent is the free state, not a broken lock

What makes this one hard to catch is that nothing looks wrong afterwards. The lock's whole implementation is "does this directory exist." An absent directory is indistinguishable from a session that finished cleanly seconds ago. There is no error, no stack trace, no log line that says a lock was stripped rather than released. You would have to already suspect the fault to go looking for it, which is exactly backwards from how you want a safety mechanism to fail.

I only caught it because I was staring at file timestamps for an unrelated reason at the exact moment it happened, and the arithmetic did not add up: a lock reading as free, next to four files with mtimes minutes old, next to the holder's own message describing itself as still working.

A correct rule was already written down for a related situation: never message a session that is mid-save. It had been measured and recorded weeks earlier. It did not save anyone here, because it was written for one shape of coordination, one coordinator messaging several workers, and this was a different shape, one session asking a peer directly. The rule was true and in the wrong container, which fires exactly as often as a rule that was never written at all.

A second fault sitting underneath the first

Chasing this down further, the save-in-progress lock turns out not to be the whole story. Part of what a session saves lives in a location that is a symlink into the shared store, and the code path that writes through that symlink never acquires the lock at all. So "the lock is free" was never a sound proxy for "nobody is writing," even before the message-strips-the-lock bug. It was an unsound proxy that had been getting lucky.

That is the uncomfortable shape of this whole investigation. Every layer I checked to confirm a session was safe to disturb turned out to be checking something adjacent to the real question rather than the question itself. The lock answers "did anyone call acquire." It does not answer "is anyone writing." Those usually agree. They do not have to.

A second live instance in the same research session made the point again from another angle: a lock with a heartbeat frozen at the moment it was taken, held by a process ID that no longer existed. The obvious read is a stale lock left behind by a crash, safe to clear. The actual explanation was that the holding process was a short-lived helper that had exited normally after acquiring the lock on the session's behalf, so a dead PID here proved nothing at all. The obvious staleness check was itself unsound, in a session where I was actively trying to be careful about exactly this class of mistake.

Never ask, only tell

The fix is a protocol that runs one way in each direction. A session waiting to save subscribes silently: it watches for the session ahead of it to release, without ever sending it a message. A session that has finished tells, explicitly and only once nothing of its own is still at risk. Asking is removed from the vocabulary entirely, because the question itself is the failure mode, not a careless way of asking it.

That inversion is the whole lesson, and it is a good one precisely because the wrong design is the one everyone reaches for first, coordination-by-polling, including the version of this I built the first time. Politely checking in feels like the responsible way to avoid stepping on someone's work. Here, the check-in is the thing doing the stepping.

One more trap worth naming, because I fell into it in the same session. The lock file stamps its start time in UTC. Reading a stamp of 07:21 against a British-summer-time wallclock of 08:21 looks like an hour of staleness that is not there; the lock was five minutes old. Compute the difference in the same timezone rather than eyeballing two clocks against each other, or a perfectly healthy lock will read as abandoned.

If you are wiring any kind of handover between concurrent AI sessions, and it very likely comes up the moment you run more than one at a time, check what your "are you done" message actually does to the session receiving it. Mine was quietly disarming the exact protection it existed to ask about.