Crash recovery

When a worker process dies, or the engine node that dispatched its work dies, the run it was executing is left behind. Nothing tells Ironflow this happened — a crashed process sends no goodbye. Recovery is therefore driven by expiry: the dead party stops refreshing a clock, the clock runs out, and a surviving node reclaims the work.

An HTTP pull worker is the exception to the node half. Its registration and the leases it holds live in the database, and its heartbeat on any surviving node extends those leases, so a running job outlives the node that handed it out (ADR 0138). Only a job whose poll response, ack or result was lost with the node is left behind, and it goes through the lease expiry below.

Three independent loops do this. They cover disjoint states, and the split is deliberate — this document explains what each one owns, why they cannot double-recover the same run, and what recovery latency to expect.

For the user-facing walkthrough (kill a worker with kill -9, watch it resume), see Survive a Crash. This page is the mechanism underneath it.

What a running segment holds

A segment is one execution attempt at a run. While it executes, it holds up to three markers in the database:

  segment admitted → dispatched → executing
        │                │            │
        │                │            └─► concurrency_lease   "I hold a capacity slot"
        │                └──────────────► dispatch_queue row  "this segment is assigned to me"
        └───────────────────────────────► worker_session      "this worker is alive"   (streaming worker)

  worker heartbeats every 30s ──► pushes every expiry clock forward

A live worker keeps pushing those clocks. A dead one does not. Every recovery decision below is a query for a clock that has passed.

Loop 1 — the capacity scanner

Owned by internal/capacity. It sweeps every 30 s (ScannerInterval) and reclaims four things:

Expired thing	What it means	What the scanner does
Reservation (ReservationExpiry, 30 s)	The worker died after being admitted but before it began executing	Frees the reserved slot
Lease (LeaseExpiry, 90 s)	The worker died mid-execution	Frees the slot and re-enqueues the segment under a fresh execution_seq, after RecoveryGrace (30 s)
Worker session (heartbeat expiry)	A streaming worker, or the node its stream ran on, is gone	Deletes the session, cascading to its credits so the dead worker stops holding phantom slots
HTTP pull worker registration (90 s without a heartbeat)	An HTTP pull worker is gone	Deletes the registration; the worker’s next poll or heartbeat gets 404 and the SDK registers again

Deleting a dead worker session does not requeue that worker’s in-flight segments. Those hold concurrency leases on the same heartbeat clock, and the lease-expiry row above re-enqueues them. Because expiry is decided by database time, a dead node’s sessions are reaped by any surviving node’s scanner — node death needs no separate code path.

Execution fencing

The fresh execution_seq is the load-bearing detail. It is a fence, not a bookkeeping counter.

  node A: segment execution_seq=3 ──► ✗ freezes (long GC pause, network partition)
                                          │
  lease expires, scanner recovers          │  ...node A is not actually dead
  node B: segment execution_seq=4 ──► executing
                                          │
  node A wakes up, writes with seq=3 ◄─────┘
        └─► REJECTED — stale seq

Without the fence, “the worker looks dead” and “the worker is dead” would be the same decision, and a frozen node resuming after a partition would corrupt a run that another node had already taken over. Staleness is enforced at the ActivateLease chokepoint, so a stale segment cannot even re-enter execution.

A run that keeps failing this way is not retried forever: after RecoveryCeiling (10) consecutive recoveries the run terminates with EXECUTION_RECOVERY_EXHAUSTED.

A resume has the same kind of ceiling. Each SDK call gets its own step ID, so a healthy run does not resume from one step many times in a row. If the engine resumes a run from the same step more than 10 times in a row, the worker keeps yielding a step that the engine already finished. The engine then fails the run with RESUME_NO_PROGRESS and does not dispatch it again. The count is on the run row, so it holds across nodes and restarts. An operator resume sets it back to zero.

Loop 2 — the stale-claim sweep

Owned by the scheduler (internal/engine/scheduler.go). It sweeps every 60 s (IRONFLOW_STALE_CLAIM_RECOVERY_INTERVAL) with a 2 min threshold (IRONFLOW_STALE_CLAIM_THRESHOLD).

The capacity scanner can only see leases, reservations and sessions. There is a state that holds none of them: a step that has been claimed for wake-up but has not yet re-entered the capacity path.

Three claim sweeps put a step into waking:

  step.sleep(...) elapsed          ──► status 'sleeping' → 'waking'
  step.waitForEvent(...) timed out ──► status 'waiting'  → 'waking'
  step.invoke(...) timed out       ──► status 'waiting'  → 'waking'
                                            │
                                            ├─ claimed_by = this node
                                            └─ claimed_at = now

Each is an UPDATE ... FOR UPDATE SKIP LOCKED in PostgreSQL, so exactly one node claims each row and no step is processed twice.

If the claiming node dies in the window between claiming the step and dispatching it, that step holds no lease and no queue row. The capacity scanner is blind to it. The stale-claim sweep resets it — status returns to its pre_claim_status, and the ordinary sleep/wait claim loops pick it up on their next tick.

The same sweep also resets two other lease-less states: a running step whose parent run’s claim has gone stale (the engine’s stale-running timeout: 5 min by default, 45 s under --dev, no environment variable — separate from the 2 min claim threshold above), and an orphaned running legacy pull run whose claim went stale, which it re-admits through the capacity path as a new segment.

Loop 3 — the orphan-admission sweep

Owned by the engine (internal/engine/orphan_sweep.go). Every node runs it once at boot and then every 30 s (the capacity scanner interval).

Every way of starting a run (emit, invoke, cron, debounce, a child invoke) commits the run in waiting_for_capacity first and writes its first dispatch_queue row a moment later. If the node dies between the two, the run holds no lease and no queue row, and its status is not one the stale-claim sweep reads. The client may already have its 200.

The sweep picks up a waiting_for_capacity run that is older than 60 s (twice the 30 s dispatch timeout), is still on its first execution_seq, and holds no queue row and no lease. On PostgreSQL the age is measured on database time, the clock that stamped the run, so node clock skew cannot shorten it.

• It re-admits the run to the dispatch queue on execution_seq 1, and the capacity dispatcher runs it as normal.
• It cancels a child invoke whose parent has already ended (parent-ended).
• It cancels a child invoke that no parent step links (invoke-link-failed), because the parent’s retry creates its own child. If the parent’s link lands after that cancel, the parent’s invoke step fails with the cancelled child instead of waiting for it.
• It fails a run whose function or triggering event is gone, or that cannot be admitted: no endpoint, an unresolvable concurrency or throttle key, a pull run with no function version.

A sweep reads up to 100 runs, oldest first, and the next starts where it stopped. So a run it keeps skipping, such as an invoked child whose cancel_on replay keeps failing and so is never dispatched, cannot hide the runs behind it. After the last page it starts again from the oldest. Every fifth sweep instead reads the runs behind that walk, with a position of its own, from the oldest run up to where the walk stood when this lap began. So newer orphans that keep every page full cannot keep it away from a run it skipped. A lap takes about 2.5 minutes at the default interval for each 100 runs still waiting behind the walk when it begins, and a run that has recovered comes back within two laps. So the wait grows with the number of runs the sweep keeps skipping. These sweeps also slow the walk by a fifth.

Every one of those writes is conditional on the run row: the run must still be waiting_for_capacity at the version the sweep read, with no queue row and no lease, and a cancel re-checks the parent link or the parent’s status under the same row lock (on PostgreSQL in a statement after the lock is taken, on SQLite inside the single guarded write). Every creation path’s own first admission is guarded too: it writes its queue row only while the run is still waiting_for_capacity with no lease, under a lock on the run row. So is its failure report: a dispatch failure reported late does not fail a run that has been queued since. Admission also bumps the run’s version, so any write based on a read from before it (a stalled key-resolution failure, for example) loses its version check. A dispatch that stalled past the sweep, a second node’s sweep, or a stale list therefore never runs the work twice. The 60 s age is not configurable.

Why the loops never collide

The stale-claim sweep’s statements carry NOT EXISTS guards against both concurrency_leases and dispatch_queue, so it will not touch a run the capacity layer owns — including the grace window after an expired lease is deleted but before its segment re-activates, when the run holds a queue row and no lease.

  ┌─────────────────────────────┬──────────────────────────────┐
  │ holds lease or queue row    │  holds neither               │
  │                             │                              │
  │   → capacity scanner        │    → stale-claim sweep       │
  │     (loop 1)                │      (loop 2)                │
  └─────────────────────────────┴──────────────────────────────┘
              disjoint by construction — never double-recovered

The orphan-admission sweep takes only waiting_for_capacity runs with neither a lease nor a queue row. The stale-claim sweep acts on claimed steps and running runs, so the two never see the same run.

The exclusion is in the SQL, not in a timing assumption. Tightening one loop’s threshold cannot cause another to steal its work.

Resetting a step is likewise a SQL-level restore: status is set back to the pre_claim_status captured when the step was claimed, and claimed_by / claimed_at are nulled — so a reclaimed step returns to sleeping or waiting, not to a generic pending state.

Recovery latency

Both loops land in the same range, because both are gated on an expiry clock plus a scan interval.

Lease expiry (loop 1) — the path a killed worker takes:

  worker dies
    └─ 60–90 s  before its lease expires  (LeaseExpiry 90 s, minus up to
    │                                      30 s already burnt since the
    │                                      last LeaseRefresh)
    └─  0–30 s  until the scanner sees it (ScannerInterval 30 s)
    │             └─► run flips 'running' → 'waiting' HERE
    └─ 30 s     recovery grace            (RecoveryGrace 30 s)
    └─  0–30 s  until the pull dispatcher's next sweep hands it
                to a worker (no kick — the scanner does not wake it)
                ─────────────────────────
                90 s to 3 min from kill to resume

Measured on a --dev SQLite server via make demo-agent-crash-resume: 115 s from kill -9 to the recovery audit pair, 148 s to the run completing.

The run reaching waiting is not the finish line — the first thing recovery does is arm a 30 s grace timer, so a live, connected, idle worker legitimately sits there doing nothing for another 30–60 s. Nothing is logged and no run field counts this down. The segment’s eligible_at is the countdown — see Observing recovery for how to read it.

Stale-claim sweep (loop 2): a 2 min threshold plus a 60 s tick gives 2–3 minutes.

--dev does not speed up a worker crash

--dev lowers only the loop 2 numbers (5 s tick, 45 s threshold). The loop 1 timings above are compile-time constants in capacity.DefaultConfig() — there is no flag, no YAML field and no environment variable for them, and --dev leaves them untouched.

A killed pull worker holds a concurrency lease, so it recovers through loop 1. That takes 90 s to 3 min on a --dev server exactly as it does in production, and IRONFLOW_STALE_CLAIM_THRESHOLD will not shorten it — the stale-claim sweep’s NOT EXISTS guard skips any run holding a lease. Budget the full wait in any local crash demo.

Any threshold you tune must stay comfortably above the 30 s heartbeat interval. Set it too low and you will reclaim work from workers that are alive and merely slow — the execution fence keeps that safe, but you pay for the same step twice.

Observing recovery

Recovery is audited. Three audit events are emitted by the capacity scanner and the orphan-admission sweep and record unconditionally — they ignore the per-function audit toggle, because they are platform concerns a tenant must not be able to suppress:

• capacity.lease.expired — a lease expired and was fenced and reclaimed
• capacity.segment.recovered — the segment was re-enqueued under a fresh execution_seq
• capacity.run.readmitted — the orphan-admission sweep re-admitted a run that was never queued

The orphan-admission sweep also counts every run it settles in ironflow_capacity_orphan_runs_total, labelled outcome = readmitted, failed or cancelled.

See Audit logging for the field list.

To inspect live state:

ironflow capacity leases     # active (unexpired) concurrency leases
ironflow capacity sessions   # pull-worker sessions
ironflow capacity queue      # queued segments — `eligible_at` is the recovery-grace countdown

These three need a platform principal (ifplatform_), not an org API key. A serve --dev server does not issue one — its bootstrap key is an org admin key (ifkey_) and returns 403 platform credentials required here. See the capacity CLI reference for how to get one.

What this does not cover

All three loops are database-driven. They recover work after a worker or an engine node dies, while the rest of the cluster and its database survive.

NATS redelivery is not a third loop for run execution. JetStream AckWait redelivery does protect the ingest edge — if a node dies after an event is accepted but before its run is created, the EVENTS-stream consumer redelivers and a surviving node routes it. Once a run exists, however, recovery is entirely lease- and claim-driven as described above. Pull-mode dispatch is an HTTP long-poll against the database, not a NATS work queue, so there is no per-step redelivery timer to wait on.

Recovery across an engine crash — every node down at once — additionally requires PostgreSQL and persistent NATS storage. An unconfigured ironflow serve (SQLite plus embedded NATS, with or without --dev) covers worker-crash recovery only. See Docker Compose.

For what happens after a run reaches an outcome, and what recovers a frame or a parent wake lost after that commit, see Run outcomes after commit.

For what your application loses while no engine is reachable at all — and which of those losses are permanent — see When Ironflow is unavailable.