Alert on the change, not the state

A fleet of sixty unattended jobs ran chronically red — not because things were failing, but because each job sensed a value, decided red or green, and pushed to a phone, all in one stateless breath. So the operator muted it. A muted alarm is a dead alarm.

The weakness

The board said 54 of 61 green and it had said something close to that for months. Almost none of the seven reds were real failures. They were settled conditions the fleet couldn't stop re-announcing, normal variance tripping a fixed line, and platform state that had gone stale hours ago. The alerts got noisy enough that the operator turned them off entirely — which is the worst outcome, because now the one real failure has nowhere to land.

The root cause wasn't any single job. It was the shape of every job. Each one did three things at once and kept no memory of having done them: it sensed a value, it decided red or green, and it pushed to the phone. Sense, judge, and page fused into one stateless act, sixty times over.

The through-line: the alerting had no memory, no tolerance for variance, and no resilience layer. It was alert-fatigue by construction — the noise wasn't a misconfiguration, it was the design working as built.

The wrong fix (the one that's always tempting)

The obvious move is to tune the thresholds. Widen the grace window a bit. Raise the quota alarm. Add another dedup key. We could have spent a week nudging constants and the board would have looked calmer for a fortnight.

It's rearranging deck chairs. Tuning treats symptoms, and the noise always grows back, because the disease isn't the values of the thresholds — it's that sensing and deciding are fused in one stateless place. You cannot tune your way out of a system that has no memory. A better constant is still a constant with amnesia.

The reframe

An alarm that fires on state is noise by construction. An alarm should fire on change. The move is to separate the sensor from the verdict: let sensors emit raw facts and nothing else, and pull all judgement into one stateful decision layer — one that has memory, tolerates variance, and heals what it safely can — so the only thing that ever reaches a human is the new, stuck, and actionable.

A value being red is not news. A value turning red, and staying red after everything safe has been tried, is news.

The fix — the alert brain

We split the fleet in two. The jobs became dumb sensors: they post a raw signal to a shared sink and make no judgement at all. One central evaluator — the brain — owns every decision. It has six parts, each aimed at one of the five causes above.

• A declarative check registry. Adding a check is one line that inherits all the behaviour below — a control plane, not logic scattered across sixty scripts. The floor rises for every check at once.
• A state machine with real memory. Each check walks green → concern → healing → confirmed → {acknowledged | recovered | paused | partial}, tracking first-seen, last-change, and ack-until. The board now knows the difference between "red right now" and "red, and I already told you at 06:00."
• A variance layer. Each metric is compared to its own rolling history — a robust median-and-MAD band, not a fixed constant — with adaptive grace for freshness. Normal wobble stays green; a genuine departure stands out.
• An idempotent-only self-heal step. Before escalating, the brain retries, re-probes, or re-deploys the last known-good. This is also where the sticky-state false-red dies: the heal step re-reads the fresh stamp before it believes the red, so a job fixed two hours ago clears itself instead of cascading.
• Rule-based auto-acknowledgement. Known-standing conditions — the settled overage, a paused-by-design channel — are acknowledged by rule with a TTL, so they sit quiet until they actually change, then speak up again.
• Escalation as the only path to the phone. Confirmed, actionable, new. Everything else lives on the board and never buzzes.

And the brain itself is a monitored job with an independent watchdog — so the thing that decides whether to page you can't quietly become a silent single point of failure. A decider that can die unnoticed is just stale-green wearing a smarter hat.

The proof

The shadow week is the honest part. A new alerting layer that starts paging on day one is asking to be muted all over again. So it runs dark first: it makes every verdict, performs every heal, and writes it all to the log — but the phone stays quiet. Only once a week of its judgement matches reality does it get the interrupt. The interrupt is earned, not assumed.

The principle, generalised

An alarm that fires on state, not change, is noise by construction.

Give a monitoring system memory, let it heal what it safely can, and spend the human's attention only on what is new, stuck, and theirs. Everything else belongs on a board, not a phone. The interrupt is a privilege the alert layer has to earn — every buzz that wasn't news spends down the trust that the next real failure depends on.

1. Separate the sensor from the verdict. Sensors emit raw facts; one stateful layer judges. Fusing them is how you get amnesiac alarms.
2. Fire on the transition, not the level. Track first-seen and last-change, so "still red" and "newly red" are different events.
3. Compare a metric to its own history, not to a fixed line. Normal variance should never buzz.
4. Heal before you escalate — idempotent-only. Retry, re-probe, re-deploy-last-good. Never mutate durable state to make a red go away.
5. Acknowledge the known and the paused. A standing condition, ack'd with a TTL, is silent until it changes — then it speaks again.
6. Make the decider observable. The brain is a monitored job with its own watchdog, or it's a silent single point of failure.
7. Earn the interrupt in the shadows. Run dark for a week before you're allowed to buzz. Trust is spent by every false page.