Survive a Crash

Ironflow agents are durable — every tool call is memoized as a recorded step, every memory write goes through an event-sourced entity stream. Crash the worker mid-run, restart it, and the agent picks up from the last completed step instead of starting over.

This tutorial walks through the literal demo we run for make demo-agent-crash-resume. Build a 3-step document-processing agent, kill it mid-pipeline, watch it resume.

What survives the crash

Agent state	Where it lives	After kill -9?
Completed step output	Step memoization table (steps)	✅ replayed from cache
Memory writes	Entity stream (agent-memory)	✅ projection re-derives
In-flight tool call	Process memory only	❌ re-runs from start
LLM tokens spent	Provider-side	❌ already paid for the partial call

You only re-run the step that was in flight when the worker died. Everything before it is replayed from the cache.

Single-binary durability scope

Single-node SQLite (the default serve --dev mode) covers worker-crash recovery, including the path that lets a fresh worker pick up an orphaned run after the worker that started it dies — the capacity scanner’s lease-expiry loop, which the SQLite store implements alongside PostgreSQL. It is gated on a lease clock, so it is not instant; see the wait callout below. Full durability across server crashes requires PostgreSQL plus persistent NATS storage — see the Docker Compose guide. Production deployments of Ironflow use Postgres + multi-node clustering.

Prerequisites

• Ironflow server running locally — ironflow serve --dev at http://localhost:9123
• Node.js 22+ and pnpm (TypeScript), or Python 3.10+ (Python)
• IRONFLOW_SERVER_URL exported in the terminal that runs the worker — set -x IRONFLOW_SERVER_URL http://localhost:9123 in fish, export IRONFLOW_SERVER_URL=http://localhost:9123 in zsh or bash. Set it even though it looks optional: createWorker would default to that URL on its own, but the agent’s memory client builds its own HTTP client from IRONFLOW_SERVER_URL and has no localhost default. Without it, it can only find the server through the .ironflow/engine.json discovery file that serve writes, and that file is found only when the worker runs in or below the directory where serve started. When no URL is found, the TypeScript SDK throws “memory() requires a runtime backend”. Skip it and Steps 1–7 can look fine right up until memory.append() in Step 8.
• Four terminal windows: one for the server, then terminals A (worker), B (emit) and C (kill) in Steps 6 to 8

Step 1 — Scaffold the agent

The full code is in examples/agents/doc-processor-agent/. To follow along inline:

mkdir -p crash-demo/src && cd crash-demo

package.json:

{
  "name": "crash-demo",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "tsx watch src/worker.ts",
    "start": "tsx src/worker.ts"
  },
  "dependencies": {
    "@ironflow/node": "*",
    "zod": "^4.4.3"
  },
  "devDependencies": {
    "tsx": "^4.19.0",
    "typescript": "^5.6.0",
    "@types/node": "^22.0.0"
  }
}

pnpm install

Step 2 — Define the tools

src/tools.ts (Python version below):

import { defineTool } from "@ironflow/node/agent";
import { z } from "zod";

// Slow OCR — simulates a real API. The 3-second sleep is the kill window.
export const ocr = defineTool({
  name: "ocr",
  description: "Extract text from a document image",
  input: z.object({ imageUrl: z.string().url() }),
  handler: async ({ imageUrl }) => {
    console.log(`[ocr] extracting text from ${imageUrl} (3s)...`);
    await new Promise((r) => setTimeout(r, 3000));
    return { text: `Extracted text from ${imageUrl}`, pages: 1 };
  },
});

export const classify = defineTool({
  name: "classify",
  input: z.object({ text: z.string() }),
  handler: async ({ text }) => {
    console.log(`[classify] categorizing ${text.length} chars...`);
    await new Promise((r) => setTimeout(r, 200));
    return { category: text.includes("invoice") ? "invoice" : "other" };
  },
});

export const publish = defineTool({
  name: "publish",
  input: z.object({ docId: z.string(), category: z.string() }),
  handler: async ({ docId, category }) => {
    console.log(`[publish] doc ${docId} → ${category} channel`);
    return { published: true };
  },
});

The same tools in Python (tools.py). define_tool takes a JSON Schema for input_schema and passes it to the model. Install ironflow-py[validate] to also validate the args against it before the handler runs. The handler receives the args dict:

import asyncio

from ironflow.agent import define_tool


# Slow OCR — simulates a real API. The 3-second sleep is the kill window.
async def _ocr(args):
    print(f"[ocr] extracting text from {args['imageUrl']} (3s)...")
    await asyncio.sleep(3)
    return {"text": f"Extracted text from {args['imageUrl']}", "pages": 1}


async def _classify(args):
    print(f"[classify] categorizing {len(args['text'])} chars...")
    await asyncio.sleep(0.2)
    return {"category": "invoice" if "invoice" in args["text"] else "other"}


async def _publish(args):
    print(f"[publish] doc {args['docId']} → {args['category']} channel")
    return {"published": True}


ocr = define_tool(
    name="ocr",
    description="Extract text from a document image",
    input_schema={
        "type": "object",
        "properties": {"imageUrl": {"type": "string"}},
        "required": ["imageUrl"],
    },
    handler=_ocr,
)
classify = define_tool(name="classify", handler=_classify)
publish = define_tool(name="publish", handler=_publish)

Step 3 — Memory projection

src/memory.ts:

import { createProjection } from "@ironflow/node";

interface DocState {
  docId: string;
  status: "ocr" | "classified" | "published";
  category?: string;
}

export const docMemory = createProjection({
  name: "doc-processor-memory",
  events: ["doc.processed"],
  initialState: (): Record<string, DocState> => ({}),
  handler: (state, event) => {
    const data = event.data as DocState;
    return { ...state, [data.docId]: data };
  },
});

The same projection in Python (memory.py):

from ironflow.projection import create_projection

doc_memory = create_projection(
    name="doc-processor-memory",
    events=["doc.processed"],
    initial_state=lambda: {},
    handler=lambda state, event, ctx: {**state, event.data["docId"]: event.data},
)

Step 4 — Wire the agent

src/agent.ts (Python version below):

import { agent } from "@ironflow/node/agent";
import { ocr, classify, publish } from "./tools.js";

interface DocReceived {
  docId: string;
  imageUrl: string;
}

export const docProcessor = agent(
  {
    id: "doc-processor",
    description: "Durable OCR → classify → publish pipeline",
    triggers: [{ event: "doc.received" }],
    memory: {
      streamId: "agent-memory",
      projection: "doc-processor-memory",
    },
    recording: true,
  },
  async ({ event, tool, memory, logger }) => {
    const { docId, imageUrl } = event.data as DocReceived;
    logger.info(`processing ${docId}`);

    // Each tool() call is a memoized step. Crash here and the
    // worker resumes from the step that was in flight.
    const { text } = await tool(ocr, { imageUrl });
    const { category } = await tool(classify, { text });
    await tool(publish, { docId, category });

    // Memory write is also durable — appends to an entity stream
    // and waits for the projection to catch up before returning.
    await memory.append("doc.processed", {
      docId,
      status: "published",
      category,
    });

    return { docId, category };
  },
);

The same agent in Python (agent.py). @agent returns a normal worker Function; the handler gets an AgentContext with ctx.tool() and ctx.memory:

from ironflow.agent import MemoryConfig, agent

from tools import classify, ocr, publish


@agent(
    id="doc-processor",
    description="Durable OCR → classify → publish pipeline",
    triggers=[{"event": "doc.received"}],
    memory=MemoryConfig(stream_id="agent-memory", projection="doc-processor-memory"),
    recording=True,
)
async def doc_processor(ctx):
    doc_id = ctx.event.data["docId"]
    ctx.logger.info(f"processing {doc_id}")

    # Each ctx.tool() call is a memoized step. Crash here and the
    # worker resumes from the step that was in flight.
    extracted = await ctx.tool(ocr, {"imageUrl": ctx.event.data["imageUrl"]})
    result = await ctx.tool(classify, {"text": extracted["text"]})
    await ctx.tool(publish, {"docId": doc_id, "category": result["category"]})

    # Memory write is also durable — appends to an entity stream
    # and waits for the projection to catch up before returning.
    await ctx.memory.append(
        "doc.processed",
        {"docId": doc_id, "status": "published", "category": result["category"]},
    )

    return {"docId": doc_id, "category": result["category"]}

Step 5 — Worker entrypoint

src/worker.ts (Python version below):

import { createWorker, type IronflowProjection } from "@ironflow/node";
import { docProcessor } from "./agent.js";
import { docMemory } from "./memory.js";

// createWorker reads IRONFLOW_SERVER_URL and
// defaults to http://localhost:9123. The agent's memory client reads the same
// variable but has no default, so set it (see Prerequisites). This also
// lets the demo script target a non-default port without editing this file.
const worker = createWorker({
  functions: [docProcessor],
  projections: [docMemory as IronflowProjection],
});

worker.start().then(() => {
  console.log("doc-processor worker ready (pid", process.pid + ")");
});

The same entrypoint in Python (worker.py). Worker.run() blocks until the worker stops. When you pass no server_url, it reads IRONFLOW_SERVER_URL:

from agent import doc_processor
from memory import doc_memory

from ironflow.worker import Worker

if __name__ == "__main__":
    Worker(functions=[doc_processor], projections=[doc_memory]).run()

For Python, install the SDK with pip install ironflow-py and run the worker with python worker.py wherever Steps 6 to 8 say pnpm dev. The kill command in Step 7 then targets python.*worker.py.

For Go, the github.com/sahina/ironflow-go/ironflow/agent package has the same primitives (DefineTool, Agent, Approve). Its README has a worked agent. The crash and resume behavior is the same, because it comes from the server.

Step 6 — Run the happy path

In terminal A start the worker:

pnpm dev

Output:

[ironflow-worker] Starting worker worker-… with 1 functions
[ironflow-worker] Registered function: doc-processor
[ironflow-worker] Connected to server at http://localhost:9123
[ironflow-worker] Projection runner started (streaming): doc-processor-memory

The doc-processor worker ready (pid …) line in worker.ts sits inside worker.start().then(...), and start() is the poll loop — it resolves only on worker.stop(). Connected to server at … is the line that means the worker is live.

In terminal B trigger a run:

ironflow emit doc.received \
  --data '{"docId":"doc-1","imageUrl":"https://example.com/invoice.png"}'

Terminal A logs:

[ocr] extracting text from https://example.com/invoice.png (3s)...
[classify] categorizing 51 chars...
[publish] doc doc-1 → invoice channel

Run completes in about 3.2 seconds. Confirm with ironflow run list or curl -H "Content-Type: application/json" -d '{}' http://localhost:9123/ironflow.v1.IronflowService/ListRuns | jq '.runs[0]'.

Step 7 — Crash mid-run

Trigger a fresh run and kill the worker during the 3-second OCR sleep:

# Terminal B
ironflow emit doc.received \
  --data '{"docId":"doc-2","imageUrl":"https://example.com/receipt.png"}'

Terminal A logs the start:

[ocr] extracting text from https://example.com/receipt.png (3s)...

Now in a third terminal — use kill -9 (SIGKILL), not Ctrl+C. SIGINT triggers a graceful drain that lets active jobs finish (up to 30 s by default); only an ungraceful kill leaves the run orphaned, which is what we want to demonstrate:

# Terminal C
kill -9 $(pgrep -f "tsx.*worker.ts")

Terminal A:

[1]  + 41203 killed     pnpm dev

The OCR call was in flight. There is no completed step recorded for it. ironflow run list shows doc-2 as running (orphaned).

Step 8 — Restart and resume

# Terminal A
pnpm dev

Output — the connect line lands right away, then nothing; the worker idles for 90 s to 3 min before the orphaned run comes back to it (see callout below):

[ironflow-worker] Connected to server at http://localhost:9123
[ocr] extracting text from https://example.com/receipt.png (3s)...
[classify] categorizing 51 chars...
[publish] doc doc-2 → other channel

The worker pulled the orphaned run, replayed any cached steps, and resumed at the in-flight one. OCR re-ran because it never completed. If we had killed the worker between OCR and classify, OCR would replay from cache and only classify + publish would re-run.

Expect a 90 s to 3 min wait — this is not a hang

The restarted worker does not pick the run up straight away, and nothing tells you why. There is no log line, no run field, and no progress bar. Wait it out.

The killed worker was mid-step, so it held a concurrency lease. Recovery is gated on that lease expiring:

  kill -9
    └─ 60–90 s  lease expires        (LeaseExpiry 90 s)
    └─  0–30 s  capacity scanner sees it  → run flips 'running' → 'waiting'
    └─ 30 s     recovery grace       (RecoveryGrace 30 s)
    └─  0–30 s  pull dispatcher's next sweep hands it to your worker

The running → waiting flip is the halfway mark, not the end — the grace timer starts there, so an idle, connected worker sits still for another 30–60 s afterwards. That is correct behaviour.

Measured on a --dev server: 115 s from kill -9 to the capacity.lease.expired /capacity.segment.recovered audit pair, 148 s to the run completing.

--dev does not shorten this, and neither does IRONFLOW_STALE_CLAIM_THRESHOLD. That variable tunes a different loop — the stale-claim sweep, which handles orphaned sleep/wait wake-ups and explicitly skips any run holding a lease. The lease timings are compile-time constants with no flag, YAML field or environment variable.

Any script that waits on this needs a deadline above the 3 min ceiling — make demo-agent-crash-resume uses VERIFY_TIMEOUT_MS=210000.

For the full mechanism behind these numbers — the capacity scanner’s lease-expiry loop, the stale-claim sweep, and why they cannot double-recover the same run — see Crash recovery.

Verify:

ironflow inspect $(ironflow run list --limit 1 --json | jq -r '.[0].id')

You’ll see steps tool.ocr, tool.classify, tool.publish, memory.append, memory.append.wait all marked complete.

Query the memory projection. GetProjection returns an object state as state and a non-object state as stateValue, alongside registry metadata. This projection holds an object, so the answer is in state; the filter takes whichever of the two is present:

curl -s -X POST http://localhost:9123/ironflow.v1.ProjectionService/GetProjection -H 'Content-Type: application/json' -d '{"name":"doc-processor-memory"}' | jq '.stateValue // .state'

{
  "doc-1": { "docId": "doc-1", "status": "published", "category": "invoice" },
  "doc-2": { "docId": "doc-2", "status": "published", "category": "other" }
}

Both docs show through, including the one the worker died on. (The SDK’s client.projections.get() peels the envelope automatically; raw curl does not.)

Why it works

Two pieces do the work:

1. Step memoization — tool(), llm(), memory.get() and memory.append() all wrap step.run(); approve() wraps step.waitForEvent(), and spawn() wraps step.invoke() (or step.invokeAsync() when await: false). Each call records (runId, stepId, input, output). On replay the runtime returns cached output for any step whose row exists, and only re-executes the first step without a row.

2. Entity-stream memory — memory.append() writes a doc.processed event to the agent’s entity stream (agent-memory), then waits for the named projection to catch up. The projection keys its state by docId and is rebuilt from events deterministically, so its state survives any node restart and rebuilds identically.

No new server primitives were introduced for any of this. Tools, LLM calls, approvals, spawns, and memory writes all compose over the same step.run + entity-stream surface that powers Ironflow workflows. Agents inherit crash-resume for free.

Run it with one command

The Makefile target reproduces the full demo:

make demo-agent-crash-resume

The script:

1. Starts ironflow serve --dev in the background
2. Builds the SDK and starts the doc-processor worker
3. Emits doc.received for a generated doc ID (demo-crash-<timestamp>)
4. kill -9s the worker after 1.5s (mid-OCR)
5. Restarts the worker
6. Polls the memory projection until it contains that doc ID
7. Tears down

make demo-agent-crash-resume is the check behind the agent-survives-crash narrative. It runs in .github/workflows/demo-crash-resume.yml, currently on manual dispatch (workflow_dispatch) only.

Next steps

• Code-review agent — adds llm() + approve() for human-in-the-loop gates.
• Doc-processor agent — the full version of this tutorial with real tests.
• AI Research agent — multi-step research with parallel search and summarize.
• Comparison: where Ironflow sits vs. LangGraph, Claude Agent SDK, Temporal, and others.
• @ironflow/langgraph — same crash-resume guarantees for LangGraph graphs.