Crash recovery

When a worker process dies, or the engine node that dispatched its work dies, the run it was executing is left behind. Nothing tells Ironflow this happened — a crashed process sends no goodbye. Recovery is therefore driven by expiry: the dead party stops refreshing a clock, the clock runs out, and a surviving node reclaims the work.

Two independent loops do this. They cover disjoint states, and the split is deliberate — this document explains what each one owns, why they cannot double-recover the same run, and what recovery latency to expect.

For the user-facing walkthrough (kill a worker with kill -9, watch it resume), see Survive a Crash. This page is the mechanism underneath it.

What a running segment holds

A segment is one execution attempt at a run. While it executes, it holds up to three markers in the database:

  segment admitted → dispatched → executing
        │                │            │
        │                │            └─► concurrency_lease   "I hold a capacity slot"
        │                └──────────────► dispatch_queue row  "this segment is assigned to me"
        └───────────────────────────────► worker_session      "this worker is alive"   (pull mode)

  worker heartbeats every 30s ──► pushes every expiry clock forward

A live worker keeps pushing those clocks. A dead one does not. Every recovery decision below is a query for a clock that has passed.

Loop 1 — the capacity scanner

Owned by internal/capacity. It sweeps every 30 s (ScannerInterval) and reclaims three things:

Expired thing	What it means	What the scanner does
Reservation (ReservationExpiry, 30 s)	The worker died after being admitted but before it began executing	Frees the reserved slot
Lease (LeaseExpiry, 90 s)	The worker died mid-execution	Frees the slot and re-enqueues the segment under a fresh execution_seq, after RecoveryGrace (30 s)
Worker session (heartbeat expiry)	A pull worker, or the node that owned it, is gone	Deletes the session, cascading to its credits so the dead worker stops holding phantom slots

Deleting a dead worker session does not requeue that worker’s in-flight segments. Those hold concurrency leases on the same heartbeat clock, and the lease-expiry row above re-enqueues them. Because expiry is decided by database time, a dead node’s sessions are reaped by any surviving node’s scanner — node death needs no separate code path.

Execution fencing

The fresh execution_seq is the load-bearing detail. It is a fence, not a bookkeeping counter.

  node A: segment execution_seq=3 ──► ✗ freezes (long GC pause, network partition)
                                          │
  lease expires, scanner recovers          │  ...node A is not actually dead
  node B: segment execution_seq=4 ──► executing
                                          │
  node A wakes up, writes with seq=3 ◄─────┘
        └─► REJECTED — stale seq

Without the fence, “the worker looks dead” and “the worker is dead” would be the same decision, and a frozen node resuming after a partition would corrupt a run that another node had already taken over. Staleness is enforced at the ActivateLease chokepoint, so a stale segment cannot even re-enter execution.

A run that keeps failing this way is not retried forever: after RecoveryCeiling (10) consecutive recoveries the run terminates with EXECUTION_RECOVERY_EXHAUSTED.

Loop 2 — the stale-claim sweep

Owned by the scheduler (internal/engine/scheduler.go). It sweeps every 60 s (IRONFLOW_STALE_CLAIM_RECOVERY_INTERVAL) with a 2 min threshold (IRONFLOW_STALE_CLAIM_THRESHOLD).

The capacity scanner can only see leases, reservations and sessions. There is a state that holds none of them: a step that has been claimed for wake-up but has not yet re-entered the capacity path.

Three claim sweeps put a step into waking:

  step.sleep(...) elapsed          ──► status 'sleeping' → 'waking'
  step.waitForEvent(...) timed out ──► status 'waiting'  → 'waking'
  step.invoke(...) timed out       ──► status 'waiting'  → 'waking'
                                            │
                                            ├─ claimed_by = this node
                                            └─ claimed_at = now

Each is an UPDATE ... FOR UPDATE SKIP LOCKED in PostgreSQL, so exactly one node claims each row and no step is processed twice.

If the claiming node dies in the window between claiming the step and dispatching it, that step holds no lease and no queue row. The capacity scanner is blind to it. The stale-claim sweep resets it — status returns to its pre_claim_status, and the ordinary sleep/wait claim loops pick it up on their next tick.

Why the two loops never collide

The stale-claim sweep’s statements carry NOT EXISTS guards against both concurrency_leases and dispatch_queue, so it will not touch a run the capacity layer owns — including the grace window after an expired lease is deleted but before its segment re-activates, when the run holds a queue row and no lease.

  ┌─────────────────────────────┬──────────────────────────────┐
  │ holds lease or queue row    │  holds neither               │
  │                             │                              │
  │   → capacity scanner        │    → stale-claim sweep       │
  │     (loop 1)                │      (loop 2)                │
  └─────────────────────────────┴──────────────────────────────┘
              disjoint by construction — never double-recovered

The exclusion is in the SQL, not in a timing assumption. Tightening one loop’s threshold cannot cause the other to steal its work.

Resetting a step is likewise a SQL-level restore: status is set back to the pre_claim_status captured when the step was claimed, and claimed_by / claimed_at are nulled — so a reclaimed step returns to sleeping or waiting, not to a generic pending state.

Recovery latency

Both loops land in the same range, because both are gated on an expiry clock plus a scan interval.

Lease expiry (loop 1) — the path a killed worker takes:

  worker dies
    └─ 60–90 s  before its lease expires  (LeaseExpiry 90 s, minus up to
    │                                      30 s already burnt since the
    │                                      last LeaseRefresh)
    └─  0–30 s  until the scanner sees it (ScannerInterval 30 s)
    │             └─► run flips 'running' → 'waiting' HERE
    └─ 30 s     recovery grace            (RecoveryGrace 30 s)
    └─  0–30 s  until the pull dispatcher's next sweep hands it
                to a worker (no kick — the scanner does not wake it)
                ─────────────────────────
                90 s to 3 min from kill to resume

Measured on a --dev SQLite server via make demo-agent-crash-resume: 115 s from kill -9 to the recovery audit pair, 148 s to the run completing.

The run reaching waiting is not the finish line — the first thing recovery does is arm a 30 s grace timer, so a live, connected, idle worker legitimately sits there doing nothing for another 30–60 s. Nothing is logged and no run field counts this down. The segment’s eligible_at is the countdown — see Observing recovery for how to read it.

Stale-claim sweep (loop 2): a 2 min threshold plus a 60 s tick gives 2–3 minutes.

--dev does not speed up a worker crash

--dev lowers only the loop 2 numbers (5 s tick, 45 s threshold). The loop 1 timings above are compile-time constants in capacity.DefaultConfig() — there is no flag, no YAML field and no environment variable for them, and --dev leaves them untouched.

A killed pull worker holds a concurrency lease, so it recovers through loop 1. That takes 90 s to 3 min on a --dev server exactly as it does in production, and IRONFLOW_STALE_CLAIM_THRESHOLD will not shorten it — the stale-claim sweep’s NOT EXISTS guard skips any run holding a lease. Budget the full wait in any local crash demo.

Any threshold you tune must stay comfortably above the 30 s heartbeat interval. Set it too low and you will reclaim work from workers that are alive and merely slow — the execution fence keeps that safe, but you pay for the same step twice.

Observing recovery

Recovery is audited. Two audit events are emitted by the capacity scanner and record unconditionally — they ignore the per-function audit toggle, because they are platform concerns a tenant must not be able to suppress:

• capacity.lease.expired — a lease expired and was fenced and reclaimed
• capacity.segment.recovered — the segment was re-enqueued under a fresh execution_seq

See Audit logging for the field list.

To inspect live state:

ironflow capacity leases     # active (unexpired) concurrency leases
ironflow capacity sessions   # pull-worker sessions
ironflow capacity queue      # queued segments — `eligible_at` is the recovery-grace countdown

These three need a platform principal (ifplatform_), not an org API key. A serve --dev server does not issue one — its bootstrap key is an org admin key (ifkey_) and returns 403 platform credentials required here. See the capacity CLI reference for how to get one.

What this does not cover

Both loops are database-driven. They recover work after a worker or an engine node dies, while the rest of the cluster and its database survive.

NATS redelivery is not a third loop for run execution. JetStream AckWait redelivery does protect the ingest edge — if a node dies after an event is accepted but before its run is created, the EVENTS-stream consumer redelivers and a surviving node routes it. Once a run exists, however, recovery is entirely lease- and claim-driven as described above. Pull-mode dispatch is an HTTP long-poll against the database, not a NATS work queue, so there is no per-step redelivery timer to wait on.

Recovery across an engine crash — every node down at once — additionally requires PostgreSQL and persistent NATS storage. An unconfigured ironflow serve (SQLite plus embedded NATS, with or without --dev) covers worker-crash recovery only. See Self Hosting and the failure and recovery playbooks.

For what your application loses while no engine is reachable at all — and which of those losses are permanent — see When Ironflow is unavailable.