Amber is not red

The daily restore-readiness job went red. The restore audit itself had succeeded — committed, clean, ready. What ran it red was a one-off Cloudflare hiccup on the step that just publishes the pretty page about the audit. The alarm fired for the wrong reason, and every alarm that fires for the wrong reason spends down the meaning of the alarm.

The weakness

One job, two very different things happening inside it. First the load-bearing work: audit whether the fleet could actually be restored from cold — repos, secrets, cold-data snapshot, runbook — and commit the verdict. Then the cosmetic tail: re-render and wrangler pages deploy a private dashboard about that verdict to a Cloudflare Pages surface. The audit is the signal. The deploy is just how the signal gets a nice page.

On the 08:05 run the deploy step caught a transient Cloudflare 522 — "received a malformed response from the API", a CDN blip that cleared on its own minutes later. But the deploy was wired … || RC=$?, so a failure to publish the page flipped the whole job's exit code. The restore-readiness signal — the thing you actually check to know you can rebuild — went red over a rendering hiccup, while the audit underneath it sat green and committed.

The wrong fix (the one that's always tempting)

The lazy fix is to swallow the deploy failure entirely — deploy || true — so the job never goes red on it again. That doesn't grade the failure, it deletes it: now a genuinely broken publish (bad project, revoked token, a surface that's been dark for a week) looks identical to a perfect run. You've traded a red that cries wolf for a green that lies. The other tempting fix — match the error signature, retry only on 522/5xx — adds a brittle string parser to chase a failure that costs twenty seconds once a day. Both miss the real shape: the problem isn't which failure, it's that there's only one grade of failure.

A red that fires on a cosmetic hiccup is a red you'll learn to ignore.

The reframe

The job has two outcomes worth telling apart, and it was only reporting one. Split the single red into a ladder: the load-bearing audit crashing is red; the audit succeeding but the page failing to publish after retries is amber — degraded, worth a glance, not an emergency; everything working is green. Then make the delivery survive the weather it runs in: retry the idempotent step before you ever grade it a failure.

• greenAudit ran, verdict committed, page published. Nothing to see. (Readiness itself travels in a separate ready metric — a not-ready audit is still a green run; the job did its job.)
• amberAudit ran and committed, but the publish failed after its retries. The signal is intact; only its delivery hiccuped. Worth a glance, never a page. The board shows ⚠ degraded, exit stays 0.
• redThe load-bearing work itself crashed — capture, render, or the audit. This is the only outcome that means you may not be able to restore. Exit non-zero. Red now means exactly one thing again.

The fix

Three small moves in one shell script, no new dependency:

• Retry the idempotent step. A tiny retry() helper wraps the deploy in exponential backoff — three attempts, 5s then 15s, ~20 seconds worst case. Re-deploying identical content is safe, so it retries on any non-zero rather than parsing error strings; a hard error just burns twenty seconds once a day and moves on. The 522 that started all this would now be swallowed on attempt two.
• Decouple delivery from crash-state. The old || RC=$? is gone. A survived-retry deploy failure sets a local DEPLOY_OK=false and never touches the job's exit code. The audit's success is no longer at the mercy of the CDN.
• Grade the outcome into the result line. A crash → red; audit-fine-but-publish-degraded → yellow; else green. The verdict rides the structured status line the job already emits (verdict + deploy_ok metric). The status hub already renders a fresh yellow as ⚠ degraded — so amber appeared with zero changes outside this one script.

The proof

Both paths were exercised before trusting the schedule: the amber path emits verdict:"yellow" with deploy_ok:"false" and exits 0; the green path emits verdict:"green" with deploy_ok:"true". The retry() helper was unit-checked in isolation — succeeds on the third attempt, returns the real exit code after exhausting all three — so the backoff can't silently loop or silently pass.

Grade your failures, or the alarm grades them all the same — and then it grades nothing at all.

The principle, generalised

A status signal is only worth checking if its states mean distinct things. The moment a cosmetic hiccup and a load-bearing failure share a colour, the colour stops carrying information — and the human on the other end learns to ignore it. The durable fix isn't to make the flaky step never fail (you can't) or to hide it when it does (you mustn't). It's to give failure a ladder: retry what's idempotent, decouple the cosmetic from the load-bearing, and reserve the loudest state for the one condition that actually warrants it.

1. Separate the load-bearing output from its delivery. The audit is the signal; the published page is just its courier. A courier stumbling is not the signal failing.
2. Retry the idempotent before you grade it. If re-running is safe, a transient blip should cost twenty automatic seconds, not a human's attention.
3. Decouple the flaky from the exit code. Never let the weakest, most weather-exposed step veto the strongest one via a shared || RC=$?.
4. Give failure more than one grade. Green / amber / red beats pass / fail whenever "degraded" and "dead" are genuinely different — which is almost always.
5. Keep amber fail-loud. Grading down is not hiding. Degraded must still be visibly degraded and must never be able to stamp green.
6. Reserve red for the one thing that means it. When red only ever means the load-bearing work crashed, red is worth reacting to again.