When Ironflow is unavailable

Ironflow is a server, not a library. Almost nothing the SDK exposes is computed in your process — emit, kv(), config(), secrets, entity-stream appends and projection reads are all network calls to the engine. There is no client-side cache to fall back on: sdk/js/node/src/config-client.ts and sdk/js/node/src/kv.ts hold no cache or TTL logic at all.

So the honest answer to “what happens when Ironflow is down” is not one answer. It depends on which of three things is down, and — for the third — on whether the thing you lost comes back.

  ┌──────────────────────────────────────────────────────────────┐
  │ 1. your worker process dies    → run resumes, nothing lost    │
  │ 2. one node of a cluster dies  → survivors take over          │
  │ 3. no engine is reachable      → this page                    │
  └──────────────────────────────────────────────────────────────┘

Cases 1 and 2 are Crash recovery. This page is case 3.

Sort by whether it comes back

The useful axis is not “which feature” but “is this recoverable”.

  ┌─────────────────┬──────────────────────────────────────────────┐
  │ lost for good   │  cron slots · inbound webhooks past retry     │
  │ pauses, resumes │  in-flight runs · workers · accepted events   │
  │ fails at call   │  everything else — no cache, no degraded mode │
  └─────────────────┴──────────────────────────────────────────────┘

Lost for good

Cron slots

On an ordinary restart, a scheduled-slot cursor more than two minutes behind advances to the next future slot only when it has no durably observed pending range. A two-hour outage of an hourly job can therefore lose two runs. The two-minute grace preserves a just-due slot when another scheduler joins.

Before looking up a due function, the scheduler durably protects its pending range. A new range runs from the cursor through the observing tick; retries keep that boundary so a restart cannot absorb later, unobserved downtime. A failed lookup or admission leaves that range pending across refresh, joining nodes and restart. Slots beyond protected coverage follow the ordinary downtime policy. Database admission commits the event, run and next-slot cursor together, so a crash cannot commit a run without advancing its cursor. Retained work advances at most one slot per schedule per tick.

Skipping is a defensible policy — firing a burst of catch-up runs after an outage is often worse than firing none, and most cron bodies are not written to be idempotent across a thundering herd. The skip is no longer silent (#1639): on restart the scheduler logs a warning (Cron slots produced no run; not backfilled), increments ironflow_cron_slots_skipped_total, and publishes a system.cron.{functionId}.missed frame. Durable slot-consumption records exclude intentional skips; the reporting window comes from the cron history bucket, so an outage longer than that bucket’s TTL leaves nothing to report — a long enough outage is still invisible.

If a scheduled job matters, assert on its effect — a freshness check on whatever it produces — rather than trusting that it ran.

Inbound webhooks past the sender’s retry budget

A third party posting to your Ironflow webhook endpoint gets a connection error. What happens next is entirely the sender’s policy, and it is not yours to configure: some providers retry for days, some give up after a handful of attempts, some never retry at all.

Once the sender stops, that event is gone. There is no backfill path, because the event was never in your system to queue.

  Stripe ──POST──► ✗ engine down
     │
     └─ retries on ITS schedule, for ITS budget ──► gives up
                                                      │
                                              permanently lost

This is the only failure on this page that no amount of Ironflow-side durability can fix. If a provider offers an event-replay API, that replay is your recovery mechanism — know before the outage whether yours does.

Pauses, then resumes

What	Behaviour
In-flight runs	Resume from the last completed step once an engine returns. See Crash recovery.
Pull workers	Reconnect on a loop. No configuration needed; the worker logs each attempt. A rejected credential or a terminal registration error stops the worker instead.
Already-accepted events	Durable in the outbox and JetStream. They flush when the engine returns — see the persistence caveat below.
Subscriptions	Reconnect automatically — but whether they resume depends on how you subscribed. See below.

Subscriptions resume only if you named a consumer group

A subscription that passes a consumerGroup gets a stable durable consumer on the server, owned by that group. It reconnects to the same consumer and picks up its backlog. A group deleted and created again under the same name is a new group: it starts at the moment it is created, with no backlog.

A subscription with no consumer group — the default — is a per-connection fan-out subscription. It is not durable. On reconnect it is a new subscription, delivering from that moment on, with no memory of what it missed.

  subscribe(pattern, { consumerGroup: 'billing' })  ──► durable, resumes
  subscribe(pattern, { onEvent })                   ──► fan-out, starts at "now"

If missing an event during a reconnect would be a correctness problem, name a consumer group.

That said, this matters less during a full outage than it looks: nothing can be published either, because publishing also goes through the engine. The backlog that survives is the work accepted in the moments before the engine went away.

This column assumes persistent storage

“Pauses, then resumes” is a claim about persistent storage on both sides: the database (PostgreSQL, or a SQLite file) and the JetStream store on disk. By default ironflow serve keeps JetStream in ironflow-nats beside the SQLite file, so a restarted process finds its accepted events. If you opt into memory mode (--nats-store-dir=""), the JetStream store dies with the process and the accepted-events row above collapses into “lost for good”. A lost disk does the same. PostgreSQL with embedded NATS has no default store directory: you must set one, or use an external NATS. Do not read this column as a guarantee for a single node. See Docker Compose.

Your application does not crash — it goes quiet

Push-mode functions are HTTP endpoints that Ironflow calls. When the engine is down it simply does not call them. Your service stays up, healthy, and idle. Nothing in your process fails, because nothing in your process is being asked to do anything.

This is worth knowing because it shapes your alerting: the absence of Ironflow traffic will not trip a liveness probe. Alert on work completed, not on the process being up.

Fails at the call site

Everything else throws immediately. There is no degraded mode and nothing is queued:

• emit, invoke, cancelRun
• streams.append and entity-stream reads
• kv(), config(), secrets
• getRun, listRuns, getProjection
• time-travel and audit reads

Handle these the way you handle any dependency that can be unreachable — at the boundary, with a timeout and a fallback appropriate to the call site. A failed config() read on a hot path is an outage in your service too, if you let it be.

Your tooling is down with it

The CLI talks to the engine over HTTP, and the dashboard is embedded in the same binary. ironflow inspect, ironflow secret and the dashboard are all unavailable during an outage — you are blind to exactly the runs you are worried about, at exactly the moment you want to look at them.

Plan for this: your run-state visibility during an Ironflow outage is whatever you have exported to your own observability stack, not the dashboard. See Audit logging.

The one exception

@ironflow/browser has an opt-in offline queue. Writes go to IndexedDB and flush when connectivity returns.

Its scope is narrow and worth stating precisely:

• It wraps only emit and streams.append. Every read still fails.
• It is opt-in and requires an identity.
• The rest of the client is unwrapped, on .client.

It exists for a browser in a tunnel, not for an engine that is down. If your users are on a train, it is the right tool. If your engine is down, it changes nothing for your backend.

Designing for it

Three questions worth answering before an outage rather than during one:

1. Which of your scheduled jobs would be harmful to skip? Alert on ironflow_cron_slots_skipped_total, and back it with an effect-based freshness check for outages longer than the history bucket’s TTL, where nothing is reported.
2. Which of your inbound webhook providers support replay? That answer is your only recovery path for the outage window.
3. What can you see when the dashboard is gone? If the answer is “nothing”, export audit events somewhere you control.

Note that a single-node deployment makes every case on this page reachable at once. Multi-node clustering (PostgreSQL, an external NATS server and a shared s3:// blob store) turns most engine failures into case 2 — one node dies, survivors take over — which is a different and much shorter page. See Docker Compose.