Auto-remediation

When a customer cluster on Ironflow Cloud degrades, the meta control plane attempts to recover it before paging a human. This page explains how that automation is bounded, why it exists, and what it deliberately refuses to do.

Why

Ironflow Cloud is a solo-founder operation in its first phase. A cluster that degrades at 03:00 UTC needs a defensible response that does not require waking someone up unless it has to. The bare minimum response — snapshot the cluster, restart it, check whether the restart worked — is mechanical, idempotent on snapshot + restart sagas already shipped, and bounded enough that getting it wrong loses minutes, not data.

The boundary is sharp: anything more interesting than “snapshot, restart, verify” stays in human hands.

How it fits

HealthMonitorFn (cron, every 5 minutes) polls every active cluster’s GET /api/v1/cluster/health endpoint. When a cluster posts ≥3 unhealthy probes inside a 20-minute window AND the most recent probe is unhealthy, HealthMonitorFn emits cloud.cluster.unhealthy.

AutoRemediateFn subscribes to that event. For one cluster, one trigger event, one acquire of the per-customer mutex, it runs:

snapshot → diagnose → restart → settle 2min → verify (3 probes 30s apart) → branch
                                                                              ├─ ok    → cloud.remediation.succeeded
                                                                              └─ fail  → cloud.remediation.failed

cloud.remediation.failed is the founder-page signal. PageFounderFn subscribes to it and pages the founder via Pushover priority 1; PR-4 opens a cstate status-page incident for the same event.

What it WILL NOT do

Five autonomy boundaries are enforced in code (ADR-0034 documents them; the saga’s unit tests pin them):

1. Never delete data. AutoRemediateFn is read + snapshot + restart only. There is no path through the saga that drops a row, truncates a table, or rolls a customer back.
2. Never bypass the mutex. No fast path for a “very degraded” cluster; the lock is acquired the same way every cluster acquires it.
3. Never restart without a snapshot. A failed snapshot aborts the remediation; the restart is the only recovery action, and the snapshot is the only rollback target.
4. Never remediate a remediation. A mutex held by a running AutoRemediateFn blocks a sibling AutoRemediateFn for the same customer.
5. Never retry restart. A failed restart pages the founder. The runbook is the recovery loop, not a saga retry against a half-restarted box.

These are not rules of thumb. They are testable invariants. A future PR that loosens one trips the test suite.

Trust model

The trust handed to AutoRemediateFn is bounded by two design choices.

First, the snapshot is the only safety net. If the saga aborts after the snapshot succeeded — for any reason — the founder gets a cloud.remediation.failed event carrying the snapshot ID. The cluster is restorable. If the saga aborts before the snapshot succeeds, the cluster is still in the same state it was when HealthMonitor flagged it, and no restart has happened.

Second, the mutex serialises destructive operations across the entire saga family. Provision, restart, resize, deprovision, daily-snapshot, and AutoRemediate all acquire the same per-customer key on SYS_provisioning_mutex. No two destructive sagas race for one customer’s cluster.

The combination means the worst credible failure mode is: a paged founder, an open status-page incident, a fresh snapshot in S3, and a cluster sitting in Failed status. That is a recoverable position, not an outage.

Escalation philosophy

cloud.remediation.failed is unconditional escalation. Every failure path emits it, regardless of which step failed. Snapshot timeout, restart timeout, verify probe N of 3 — all converge on the same event with a reason field carrying the specifics.

There is no “soft failure” branch. The first verify probe that fails ends the saga; the founder gets the page; the human takes over.

This is deliberate. A “let it self-heal a bit longer” branch is the kind of thing that turns a flapping cluster into a multi-hour silent outage. The single-shot saga is the right primitive; if HealthMonitor keeps flagging the cluster after the founder fixes it, HealthMonitor will trigger another AutoRemediateFn anyway.