How to think and act with a deterministic mindset

A system is deterministic when its behaviour is a function of its inputs — not of luck, ordering, or whatever ambient state happened to be lying around when it ran. The mindset is the habit of hunting down every place where that stops being true and closing it.

The idea

Close the gaps, and the thing does the same thing every time — and when it fails, it fails loud and fails the same way. That's the whole discipline.

Most of the trouble I've had with unattended systems wasn't a bug in the usual sense — the code was "correct." The trouble was a second, invisible input the code depended on and never named: a clock, an ordering, a second writer, a primitive that didn't behave the way the docs implied. Those are the sources of nondeterminism. Find them, name them, remove them.

A fixed clock is a bet on timing. Sooner or later the bet loses, and it loses quietly.

Coupling made of time

The sharpest lesson came from an editorial pipeline that froze. A consumer job ran at 09:00 every day. It read a keystone file that a producer wrote earlier each morning. Most days the producer finished around 07:00, the consumer ran at 09:00, everything was fine. Then one day the producer took nearly three hours and finished at 11:32. The consumer had already run at 09:00, found no file, 404'd, and exited. Nothing was broken. The ordering just didn't hold that day.

The 09:00 was never a schedule. It was a hope that the producer had finished first, dressed up as a schedule. The fix wasn't a better guess at the timing — it was to delete the timing bet entirely. The producer's success became the trigger. Now the keystone exists by construction at the moment the consumer runs, because the consumer only runs because the keystone was written. The race-prone cron got paused. The dependency that was implicit in the clock is now explicit in the wiring.

If two things must happen in order, make the ordering a fact of the structure, not a coincidence of the calendar. That's the same move as removing the coupling, not the component (№ 1) — you don't make the timing more reliable, you delete the dependency on timing.

One owner per output

A dashboard reverted to raw jargon overnight and nobody had changed it. It turned out two deployers were both writing the same site hourly — a local scheduled job and a cloud workflow — and whichever ran last won. The "revert" was just the older writer landing last that cycle. The same shape showed up in security headers: a duplicated x-content-type-options, because a managed transform and a custom rule were both emitting it, and the custom rule used add (append) instead of set (overwrite).

Two writers on one sink is nondeterminism by construction. The output is a function of which one ran last, and you don't control that. The fix in both cases was the same: exactly one owner per output. Retire the second deployer. Use set, not add, so the last write is the only write that matters and it overwrites cleanly. When you find behaviour that changes without an input changing, look for a second writer before you look for a bug.

A no-op must never stamp green

Stale-green is the failure that compounds, because it hides. A job that does nothing and reports success trains you to trust it, right up until the day the nothing matters. So the rule is: prove the load-bearing path first, in seconds, and die non-zero before doing any expensive work. Can I reach the sink? Is the credential valid? If not, fail loud, now — don't grind through the job and stamp green on an empty result.

The same honesty applies to what a green even means. "Last run" has to mean "last actually ran," not "last wrote output" — because a self-redirected or silent run leaves the output file untouched and the timestamp reads fresh. I had a status hub reading log timestamps and calling that "last run"; it lied whenever a run produced no new bytes. Read the authoritative result line, not the side effect.

Know your primitives

Determinism is only as good as your knowledge of what your tools actually do on the actual platform. macOS has no flock — reach for it in a lock script and you get a silent exit-0, which is stale-green wearing a lock's clothing. Use a mkdir-based lock instead, because mkdir is atomic and does exist. Guard free space with df, not du, because du over a terabyte takes ten minutes and a preflight slower than the work it guards isn't a preflight. Pick primitives by their real guarantees on the real surface, not by what should work.

The degraded state is designed, not emergent

The most deterministic systems I run aren't the ones that never degrade — they're the ones whose degradation is designed. A spend-aware editorial has an explicit ladder: a top model, then a cheaper one, then an LLM-free deterministic recap, with a documented exit code for "degraded but published." Under stress it doesn't guess and it doesn't die; it steps down a rung you already specified. A system that is either perfect or dead is less predictable than one with a known, graceful floor — that's the lesson banked as silence is a failure mode (№ 8).

Part of that honesty is refusing silent truncation. When a system bounds its coverage — top-N, a sample, no retry — it has to say so. Silent truncation reads as "covered everything" when it covered the first fifty. Determinism includes being honest about what you did and didn't do.

The moves

1. Make dependencies explicit. Never bet on timing or ordering. If B needs A, let A's success be what triggers B.
2. One owner per output. Overwrite, don't append. When behaviour drifts without an input changing, look for a second writer.
3. Fail loud, fail early, fail the same way. A no-op never stamps green. Prove the load-bearing path before the expensive work.
4. Make "success" mean what it says. "Last run" is "last actually ran," not "last wrote output." Read the authoritative signal, not the side effect.
5. Know your primitives' real guarantees on the real platform — not the ones the docs imply.
6. Design the degraded state. Give stress a rung to step down to, with a known exit code, instead of letting failure improvise.
7. Be honest about coverage. Announce what you dropped. Silent truncation is a lie the system tells with a green light.
8. Prefer reproducible-by-construction over hope-the-race-holds. If you can't reason about why it works, you don't yet own it.