Reach the source, not the courier

The fleet's new primary machine got its secrets the way you'd hand a house key to a neighbour — copied across by hand, from one specific laptop, and only if someone remembered to. It worked every day. It would fail the one day it mattered.

The weakness

A Linux box had quietly become the primary runner for the whole fleet — dozens of unattended jobs, the machine that now does the real work. But its secret-bearing env files arrived by a manual scp from one Mac. Materialise them on the laptop, copy them over, done. Undocumented, invisible, and living in nobody's runbook — a copy step that survived only in the memory of the person who first ran it.

That put a specific laptop, and a human who recalled a command, in the restore path of the fleet's most load-bearing machine. Rebuild the box, or lose the Mac, and the primary runner couldn't be reconstituted without someone remembering how. The secrets already had a source of truth — Google Secret Manager, the backbone the whole fleet uses. The box simply couldn't reach it. So a copy was couriered across by hand, and the copy, not the source, became the thing the box depended on.

The wrong fix (the one that's always tempting)

Two tempting fixes, and one lazy one. Put a real secrets agent on the box — 1Password. It died on a gate: service accounts need a Business plan the account doesn't have. Install the cloud SDK so the box can talk to Secret Manager the official way — a 150 MB toolchain, a package-manager fight on a locked-down host, and a virtualenv that rots on the next Python bump, all to make a dozen HTTPS GETs a few times a year. Or, laziest of all: just document the scp better.

A better-documented manual step is still a manual step. A heavier dependency is still a dependency that can rot. None of these delete the courier — they just dress it up. The copy is still in the path.

The reframe

Don't make the copy more reliable — delete the copy. Let the machine that needs the secrets reach the source of truth itself, with nothing between them but a scoped, read-only key. And do it with tools already on the box, so the reach adds no new thing that can rot.

A secret you can only restore by hand is a secret you've already half-lost.

The fix — the self-seeding box

The box now renders its own env files directly from Secret Manager over the plain REST API, run by hand on rebuild or rotation. One command replaces the remembered copy. Five things make it safe to leave in place:

• Templates, not values. Each env file is a checked-in template with {{sm:name}} placeholders. The mapping from secret to variable is explicit and version-controlled; no secret value ever lives in git.
• Dependency-optional auth. Minting the OAuth token prefers google-auth if it happens to be importable — but on the box it falls back to a ~25-line RS256 JWT signed with the openssl binary that's already there. The box needs python3 stdlib and openssl, nothing else: no gcloud, no google-auth, no venv, no 1Password.
• Fetch-all-before-write. The preflight pulls every secret into memory first; if any one is missing or denied, it dies non-zero without writing a single file. The box is never left half-seeded, and a no-op can never stamp green.
• Atomic writes with rollback. Each file is written temp-then-rename at 0600, keeping a .pre-materialize backup — instant reversion if a render ever looks wrong.
• Bounded blast radius. The box's service account can read exactly the thirteen fleet secrets, granted per-secret — never the shop's encryption keys, DNS tokens, or GitHub PATs. A leaked box key spends its whole blast radius on the box's own thirteen and stops there.

The proof

The cutover was shadow-first. Render to a staging directory, compare every variable to the live value by hash — never printing a secret — and confirm all eighteen are byte-identical before flipping anything. Then cut over, and let two real jobs run green on the freshly-seeded env, one of them making a live upstream API call with a key the box had just fetched for itself. The copy was proven identical to the source before the source was trusted.

Make the source reachable by the thing that needs it — with no human in the path.

The principle, generalised

A manual copy step is a coupling wearing the disguise of a convenience. It works every day until the day you actually need it, and the person who knew it is gone. The durable fix is rarely a more reliable courier — it's giving the destination a direct, scoped path to the source, and paying down, not adding to, the pile of things that can rot. The box went from a pet fed by hand to a machine that reconstitutes itself from the source of truth with one command.

1. Delete the copy, don't harden it. A better-documented manual scp is still a human in the restore path.
2. Reach the source of truth directly. The secrets already lived in one backbone; the fix was letting the box read it, not relaying a snapshot.
3. Add no dependency you'll have to maintain. Stdlib plus a binary already present beats a 150 MB SDK and a venv that rots. Every dependency refused is a failure you'll never debug.
4. Scope the key to its blast radius. Per-secret access, so a leaked key reads the box's thirteen and nothing else.
5. Fetch all before you write any. Half-seeded is worse than unseeded — die before the first write, so a no-op never stamps green.
6. Keep the bootstrap outside its own reach. The one seed key is stored, but never readable by the thing it bootstraps.
7. Prove the copy is identical before you trust it. Shadow-render, compare by hash, then cut over.