Docker Compose

Docker Compose runs the Ironflow engine container on one machine with one command: a single process, or a full cluster with PostgreSQL, NATS JetStream and an S3 store. It is the supported way to self-host Ironflow, and it also lets you test cluster behavior on your laptop.

You need Docker Compose v2.23.1 or later (docker compose version).

Choose a size

Size	File	What runs	Use it for
Single node	docker-compose.single-node.yml	1 Ironflow with embedded NATS; SQLite, or PostgreSQL with --profile postgres	Development, small apps
Starter	cluster/docker-compose.starter.yml	1 Ironflow, 1 PostgreSQL, 1 NATS, 1 SeaweedFS (S3)	Self-hosting in cluster mode on one node
Cluster	cluster/docker-compose.cluster.yml	3 Ironflow behind nginx, 1 PostgreSQL, 3 NATS, 1 SeaweedFS	An Ironflow node or a NATS server can stop without downtime
HA	cluster/docker-compose.ha.yml	3 Ironflow behind nginx, 3 PostgreSQL with automatic failover (Patroni + etcd), 5 NATS, 1 SeaweedFS	The PostgreSQL primary can also stop without manual action

Starter, Cluster and HA run Ironflow in cluster mode (--nats-url, PostgreSQL, shared S3), so you can move up a size without changing how your app connects. Run one size per host at a time: the sizes use the same host ports.

Start Starter, Cluster or HA

The compose files are in the public release repository. Each copy runs the Ironflow image of the release it came with. A clone of main gives you the newest release.

git clone --depth 1 https://github.com/sahina/ironflow-releases.git
cd ironflow-releases/deploy/docker-compose/cluster
./init-env.sh

init-env.sh writes .env with generated values for every required key (master key, JWT secret, PostgreSQL passwords, S3 keys, first admin API key, dashboard password) and prints the admin API key and the dashboard login. It needs openssl. It never overwrites an existing .env. Keep a copy of .env: without its IRONFLOW_MASTER_KEY, the secrets the stack stored cannot be read. The Starter, Cluster and HA files refuse to start while a required key is empty.

Starter

docker compose -f docker-compose.starter.yml up -d

The API and dashboard are at http://localhost:9123.

Cluster

docker compose -f docker-compose.cluster.yml up -d

nginx at http://localhost:9000 spreads requests round-robin over three Ironflow nodes. Any node serves any request, HTTP pull workers included: their registrations and job leases are in PostgreSQL. The NATS servers form a 3-node JetStream cluster, and every stream and KV bucket keeps 3 copies (NATS_STREAM_REPLICAS=3). One Ironflow node or one NATS server can stop and the cluster keeps serving.

HA

docker compose -f docker-compose.ha.yml up -d

nginx at http://localhost:9000 spreads requests over three Ironflow nodes, as in Cluster. NATS runs five servers, and every stream and KV bucket keeps 3 copies. PostgreSQL runs three Spilo nodes (PostgreSQL 18 with Patroni) and a 3-node etcd cluster that elects the primary. Ironflow lists all three PostgreSQL nodes in its connection URL with target_session_attrs=read-write, so after a failover it connects to the new primary on its own. A one-shot pg-init service creates the ironflow role and database on first start.

Each commit waits until one replica has it (Patroni synchronous mode), so a failover does not lose a write that Ironflow already acknowledged. If the primary crashes, Patroni promotes a replica once the old primary’s leader lock expires, which takes up to 30 seconds (28 seconds in our failover test). Requests that need the database fail during that window, so clients should retry. Stopping only the primary (docker compose stop pg-1) hands over at once. etcd is reachable only from the PostgreSQL nodes.

The first start pulls about 2.3 GB for the Spilo image.

Only the entry point listens on all network interfaces: 9123 for Starter, 9000 for Cluster, HA and the multi-node file. The other published ports (PostgreSQL in Starter and Cluster, NATS, SeaweedFS, the per-node Ironflow ports and their metrics ports) bind to 127.0.0.1, so you reach them from the host but not from the network. Each node serves /metrics on its API port plus 100 (9223, 9224, 9225), because that endpoint needs no credentials. The entry point answers 404 for it. In docker-compose.single-node.yml, IRONFLOW_METRICS_HOST_PORT moves the host side of that port. The other files have no such setting. HA publishes no PostgreSQL port and only the first NATS server; use docker compose exec to reach the others. If another service already uses 5432 or 8333 on the host, set POSTGRES_HOST_PORT or S3_HOST_PORT in .env.

Every service restarts on its own (restart: unless-stopped), so the stack normally comes back after a host reboot. There is one known exception: on HA, a host reset can leave a JetStream stream with no leader for good. The API stays up and /ready answers 503. See Recover a stuck JetStream stream.

The node healthchecks read /health, so a stuck stream does not stop a node from reading healthy. In Cluster and HA, ironflow-2 and ironflow-3 start after ironflow-1 answers /health. Each node waits for the NATS servers to read healthy, then starts anyway if they do not, because a NATS server fails its healthcheck while one of its streams is stuck.

In Cluster and HA, nginx reads unhealthy in docker ps when no node answers /ready through it. This healthcheck only reports: nginx sends requests to every node either way, so it can read unhealthy while the API still serves. Nothing restarts nginx, and it reads healthy again once a node answers. It can read unhealthy for a minute or two after a reboot, and for as long as a stream is stuck. docker compose up --wait fails while it does.

Start a single node

The single-node file runs one Ironflow process with embedded NATS. It does not use init-env.sh. It lives one directory up, in deploy/docker-compose/.

git clone --depth 1 https://github.com/sahina/ironflow-releases.git
cd ironflow-releases/deploy/docker-compose

With SQLite, nothing else is needed: docker compose -f docker-compose.single-node.yml up -d. SQLite supports one node only. For PostgreSQL, create a .env next to the file and start the postgres profile. The profile only starts the PostgreSQL container. Ironflow keeps using SQLite until IRONFLOW_DATABASE_URL is set; the file defaults it to empty. Set it in .env, as shown here:

.env

POSTGRES_PASSWORD=<openssl rand -hex 24>
IRONFLOW_DATABASE_URL=postgres://ironflow:<the same password>@postgres:5432/ironflow?sslmode=disable
IRONFLOW_MASTER_KEY=<openssl rand -hex 32>   # without it, secrets are stored unencrypted
IRONFLOW_JWT_SECRET=<openssl rand -hex 32>
IRONFLOW_BLOB_URL=s3://ironflow-artifacts?endpoint=https://your-s3-endpoint.example.com
AWS_ACCESS_KEY_ID=<your S3 access key>
AWS_SECRET_ACCESS_KEY=<your S3 secret key>

PostgreSQL requires explicit JWT material and S3 blob storage, even when blob offload is disabled. Create the bucket and replace the S3 endpoint and credentials above with your provider’s values. Keep the JWT key stable across restarts. The Compose command supplies the explicit bootstrap output path.

docker compose -f docker-compose.single-node.yml --profile postgres up -d
curl http://localhost:9123/health

POSTGRES_IMAGE (default postgres:16-alpine) is frozen at the major that created the postgres-data volume; a newer major refuses to start against it. To move to a newer major, see Move to a newer PostgreSQL major. To use a managed PostgreSQL instead (Supabase, Neon, RDS), leave out --profile postgres and point IRONFLOW_DATABASE_URL at it, usually with sslmode=require.

The monitoring profile adds Prometheus (:9090) and Grafana (:3000): docker compose -f docker-compose.single-node.yml --profile postgres --profile monitoring up -d. Metrics are on by default in this file. Prometheus also evaluates the 20 engine alert rules. This profile does not configure notification delivery.

Log in and create an API key

Authentication is always on.

Starter, Cluster and HA: init-env.sh printed the admin API key and the dashboard login (admin@ironflow.local and a generated password); both are in .env as IRONFLOW_BOOTSTRAP_ADMIN_KEY and IRONFLOW_BOOTSTRAP_ADMIN_PASSWORD. They are stored at the first start; change the dashboard password in the dashboard afterwards.

Single node: Ironflow writes a generated admin API key and dashboard password to a file inside the container, and the startup banner names it. Read it once, copy both values somewhere safe, then delete it:

docker compose -f docker-compose.single-node.yml logs ironflow | grep -A8 "Admin API Key"
docker compose -f docker-compose.single-node.yml exec ironflow cat /data/.ironflow/.ironflow_bootstrap_key.json
docker compose -f docker-compose.single-node.yml exec ironflow rm -f /data/.ironflow/.ironflow_bootstrap_key.json

The single-node Compose file uses /data/.ironflow/.ironflow_bootstrap_key.json for both backends. See Bootstrap key.

Then create a key for your app instead of using the admin key. Without --role the key is a read-only viewer (ironflow role list shows the roles):

export IRONFLOW_API_KEY=<admin key>
ironflow apikey create my-app --role <role-id> -s http://localhost:9123   # :9000 for Cluster and HA

Connect your app

Set IRONFLOW_SERVER_URL (http://localhost:9123 for Single node and Starter, http://localhost:9000 for Cluster and HA) and IRONFLOW_API_KEY in your app’s environment.

TypeScript

// Server-side (Node.js)
import { createClient } from "@ironflow/node";

const client = createClient({
  serverUrl: process.env.IRONFLOW_SERVER_URL || "http://localhost:9123",
  apiKey: process.env.IRONFLOW_API_KEY,
});

// Browser-side
import { ironflow } from "@ironflow/browser";

ironflow.configure({
  serverUrl: process.env.NEXT_PUBLIC_IRONFLOW_SERVER_URL || "http://localhost:9123",
  auth: {
    token: session.accessToken,
  },
});

session.accessToken should be short-lived and issued by a trusted backend. Do not expose an ifkey_ environment key through a NEXT_PUBLIC_* variable.

Go

import "github.com/sahina/ironflow-go/ironflow"

client := ironflow.NewClient(ironflow.ClientConfig{
    ServerURL: os.Getenv("IRONFLOW_SERVER_URL"),
    APIKey:    os.Getenv("IRONFLOW_API_KEY"),
})

Python

import os

from ironflow import IronflowClient, IronflowRPC

server_url = os.environ.get("IRONFLOW_SERVER_URL", "http://localhost:9123")
api_key = os.environ.get("IRONFLOW_API_KEY")

client = IronflowClient(server_url=server_url, api_key=api_key)  # REST
rpc = IronflowRPC(server_url=server_url, api_key=api_key)        # ConnectRPC

Worker and serve() read IRONFLOW_SERVER_URL and IRONFLOW_API_KEY by themselves.

To run your app and Ironflow in one compose file, see Run Ironflow alongside your app.

Pin and upgrade a version

To run a specific release, clone its tag. Find the tags on the releases page.

git clone --depth 1 --branch v<version> https://github.com/sahina/ironflow-releases.git

To upgrade, read the release notes, then check out the new tag and recreate the containers. Your .env and the volumes stay. Ironflow applies database migrations on startup.

git fetch --tags
git checkout v<new-version>
docker compose -f docker-compose.<size>.yml pull
sync
docker compose -f docker-compose.<size>.yml up -d

sync writes the new image layers to disk. Docker does not do this before pull reports success, so a host reset in the next 35 seconds or so can leave an image that does not start. See After a host reset.

If the new release changes the NATS image on Cluster or HA, do not run the last command as it is. It recreates every NATS container at once, and JetStream loses quorum. Follow Upgrade the NATS image instead.

To run another image without changing files, set a variable in .env. The Single node file reads VERSION and pulls ghcr.io/sahina/ironflow-releases:${VERSION}. The Starter, Cluster, HA and multi-node files read IRONFLOW_IMAGE, so set it to a full reference, for example ghcr.io/sahina/ironflow-releases:<version>. In the published files, the default is the release the file came with, not latest.

If a release adds a required key, Compose stops and names it: add it to .env by hand (.env.example shows each key and how to generate it). Do not run init-env.sh again. In Starter and Cluster, never change POSTGRES_PASSWORD after the first start: the database keeps the password it was created with. A Cluster stack started before init-env.sh existed used the password ironflow. In HA, pg-init sets POSTGRES_PASSWORD on every start, so you can change it; never change POSTGRES_SUPERUSER_PASSWORD or POSTGRES_REPLICATION_PASSWORD, which apply only when the cluster is created.

A Starter, Cluster, HA, or multi-node stack created before NATS_FILE_STORAGE=true keeps EVENTS, STEPS, RESULTS, PUBSUB, and CANCEL in memory. NATS does not change a stream from memory to file. Ironflow then exits at startup with stream configuration update can not change storage type. The streams stay in memory, and their messages stay, until every NATS server restarts.

Stop the Ironflow containers. Delete those five streams. Start the stack again. Leave the NATS volume in place. It holds the KV buckets, which are already on disk (secrets, config, cron, and debounce). Deleting a stream discards the messages in that stream. After every NATS server has restarted, the memory streams are already gone. Starting Ironflow then creates the file streams, and there is nothing to delete.

for s in EVENTS STEPS RESULTS PUBSUB CANCEL; do
  nats --server nats://127.0.0.1:4222 stream rm --force "$s"
done
docker compose -f docker-compose.<size>.yml up -d

Starter, Cluster, and HA publish the first NATS server on 127.0.0.1:4222. The multi-node file publishes its server on 127.0.0.1:4222 as well.

Deleting PUBSUB replaces it

A new PUBSUB is a stream your projections and consumer groups have never seen. SDK projections are refused with failed_precondition, SQL projections go to status error, and consumer group durables are rebuilt as DeliverNew. Delete and register each projection again after this step. See Bring projections back. Do not use this command for a stream that has lost its leader until you read Recover a stuck JetStream stream.

Upgrading from a release before v0.43.1: cron schedules now run in UTC, and automatic waits report waiting instead of paused. Stop every node, back up the database, then start every node on the new version; old and new versions must not share the database. See the v0.43.1 release notes.

Upgrade the NATS image

The Compose files pin nats:2.15.2-alpine and set jetstream.limits.default_max_consumers: -1. Upgrade every server to the Release A pin, 2.14.9, before moving to 2.15. Releases older than Release A used 2.12.5, which has a clustered consumer-loss bug fixed in 2.12.6. Do not jump from that pin directly to 2.15. Keep the Release A files and a verified backup for rollback.

Starter runs one NATS container, so docker compose up -d is enough. Single node embeds NATS in the Ironflow image and has no NATS image to upgrade. Cluster (3 NATS nodes) and HA (5 NATS nodes) need a rolling upgrade. Recreate one NATS node at a time, and wait until it is healthy before you start the next one. If you recreate all nodes at once, the cluster can lose quorum, and JetStream stops until enough nodes are back.

Cluster:

git fetch --tags
git checkout v<new-version>
docker compose -f docker-compose.cluster.yml pull
sync
for n in nats-1 nats-2 nats-3; do
  docker compose -f docker-compose.cluster.yml up -d --no-deps $n
  until docker compose -f docker-compose.cluster.yml exec -T $n wget -q --spider http://localhost:8222/healthz; do sleep 2; done
done
docker compose -f docker-compose.cluster.yml stop ironflow-1 ironflow-2 ironflow-3
docker compose -f docker-compose.cluster.yml up -d

HA:

git fetch --tags
git checkout v<new-version>
docker compose -f docker-compose.ha.yml pull
sync
for n in nats-1 nats-2 nats-3 nats-4 nats-5; do
  docker compose -f docker-compose.ha.yml up -d --no-deps $n
  until docker compose -f docker-compose.ha.yml exec -T $n wget -q --spider http://localhost:8222/healthz; do sleep 2; done
done
docker compose -f docker-compose.ha.yml stop ironflow-1 ironflow-2 ironflow-3
docker compose -f docker-compose.ha.yml up -d

The /healthz check passes when the node is making forward progress on its streams and consumers. It does not mean the node has no lag. A node that is not ready returns 503, which holds the loop. A server that holds a copy of a stuck stream never passes, so the loop stops there. We have not tested a NATS upgrade with a stuck stream, so recover the stream first. Stop every Ironflow node before the last command and start them together, so an old and a new Ironflow version do not run side by side. Some releases require this; the release notes say which. The Ironflow services then connect to the new NATS nodes. Do not change NATS_STREAM_REPLICAS while the NATS nodes run different versions.

To roll back from 2.15, check out the Release A tag and run the same block. Its files return NATS to 2.14.9 and remove default_max_consumers together; 2.14 rejects that field. Do not downgrade below 2.14.7. If rolling back only Ironflow while keeping NATS 2.15, keep default_max_consumers: -1 on every server first: an old Ironflow boot resets explicit stream caps to -1, which otherwise means the server default of 1,000. Prefer a fix forward.

Data and backups

Every service keeps its state on a named Docker volume, so docker compose down and up keep your data. docker compose down -v deletes the volumes and all data.

Volume	Holds
ironflow-data (Single node)	SQLite database, embedded NATS data, JWT secret
ironflow_data (Starter)	The Ironflow data directory, /data. Runs and events are in PostgreSQL, and secrets and config are in NATS, not here
postgres-data / postgres_starter_data / postgres_cluster_data	Runs, steps, events, entity streams, projections, API keys
nats_data / nats_1_data … nats_5_data	JetStream store. File-backed streams (EVENTS, STEPS, RESULTS, PUBSUB, CANCEL) and the KV buckets (secrets, config, cron, debounce). A stack created before NATS_FILE_STORAGE=true kept those five streams in memory, so the volume held the KV buckets only
pg_1_data … pg_3_data (HA)	PostgreSQL data of each Spilo node
etcd_1_data … etcd_3_data (HA)	Patroni’s record of which node is primary
seaweedfs_data	Blobs, once blob offload is on (IRONFLOW_ARTIFACT_THRESHOLD)

A working restore needs three things, and a pg_dump alone restores a server that cannot read its secrets:

• The PostgreSQL database.
• The NATS data. Secrets and config live in its JetStream KV buckets.
• .env. Its IRONFLOW_MASTER_KEY decrypts the secrets in those buckets.

The backup archive holds the master key and the PostgreSQL password next to the data they decrypt. Keep it chmod 600, encrypt it at rest, and never commit it.

Ironflow runs as a non-root user (UID 100). If you replace a named volume with a bind mount, make the host directory writable: chown -R 100:101 ./data.

Back up and restore a single node

Run these scripts from the directory that holds docker-compose.single-node.yml and .env. They assume the PostgreSQL profile. The ironflow container stops for the length of the backup, because writes made after pg_dump reads the database would be lost. Save the script and run it with bash backup.sh, so that set -e stops a failed dump before it is packaged:

#!/usr/bin/env bash
set -euo pipefail
umask 077
dc=(docker compose -f docker-compose.single-node.yml --profile postgres)
trap '"${dc[@]}" start ironflow' EXIT
out=ironflow-backup-$(date +%Y%m%d-%H%M%S)
mkdir "$out"

"${dc[@]}" stop ironflow
"${dc[@]}" exec -T postgres pg_dump -U ironflow ironflow > "$out/ironflow.sql"
grep -q 'PostgreSQL database dump complete' "$out/ironflow.sql"
cp .env "$out/env"
docker run --rm -v ironflow_ironflow-data:/data:ro -v "$PWD/$out:/backup" \
  alpine tar czf /backup/ironflow-data.tar.gz -C /data .

tar czf "$out.tar.gz" "$out" && rm -rf "$out"
echo "wrote $out.tar.gz"

The volume names carry the Compose project name, so ironflow-data is ironflow_ironflow-data to docker volume. With SQLite, delete the pg_dump and grep lines: the database is inside ironflow_ironflow-data.

To restore, start on the same VERSION the backup came from: an older engine against a migrated database is an unsupported downgrade. Then run bash restore.sh ironflow-backup-<timestamp>.tar.gz:

#!/usr/bin/env bash
set -euo pipefail
dc=(docker compose -f docker-compose.single-node.yml --profile postgres)
tar xzf "$1"
src=${1%.tar.gz}

"${dc[@]}" down
# Restore .env before anything starts, or the restored secrets stay unreadable.
cp "$src/env" .env
chmod 600 .env
docker run --rm -v ironflow_ironflow-data:/data -v "$PWD/$src:/backup:ro" alpine \
  sh -c 'find /data -mindepth 1 -delete && tar xzf /backup/ironflow-data.tar.gz -C /data'
docker volume rm -f ironflow_postgres-data
"${dc[@]}" up -d --wait postgres
"${dc[@]}" exec -T postgres psql -v ON_ERROR_STOP=1 --single-transaction \
  -U ironflow ironflow < "$src/ironflow.sql"
"${dc[@]}" up -d
rm -rf "$src"

Back up Starter, Cluster and HA

Dump PostgreSQL with the stack running and keep a copy of .env:

docker compose -f docker-compose.starter.yml exec -T postgres pg_dump -U ironflow ironflow > ironflow.sql   # also Cluster
docker compose -f docker-compose.ha.yml exec -T pg-1 pg_dump -U postgres ironflow > ironflow.sql

In HA, three PostgreSQL copies protect against a stopped node, not against a bad write or a deleted table, so back up anyway. Run the dump on any node. The dump does not hold secrets or config, which live in the NATS volumes. Keep their values in your own secret store, and set them again after a restore (ironflow secret set), as in Move to a larger size.

If you load an older dump under running engine nodes, restart every node after the load. A node keeps a policy epoch above the database’s until the database passes it or the node restarts, and until then it does not apply policy set edits.

Move to a newer PostgreSQL major

POSTGRES_IMAGE pins the major that created the PostgreSQL volume, and a newer major refuses to start against it. To move, dump the database, delete the volume, change the image, and restore. For a single node, run this as a script so that set -e stops a failed dump before the irreversible steps:

#!/usr/bin/env bash
set -euo pipefail
dc=(docker compose -f docker-compose.single-node.yml --profile postgres)

"${dc[@]}" stop ironflow
"${dc[@]}" exec -T postgres pg_dump -U ironflow ironflow > ironflow.sql
grep -q 'PostgreSQL database dump complete' ironflow.sql

"${dc[@]}" down
docker volume rm ironflow_postgres-data
sed -i.bak '/^POSTGRES_IMAGE=/d' .env
echo 'POSTGRES_IMAGE=postgres:17-alpine' >> .env

"${dc[@]}" up -d --wait postgres
"${dc[@]}" exec -T postgres psql -v ON_ERROR_STOP=1 --single-transaction -U ironflow ironflow < ironflow.sql
"${dc[@]}" up -d

Starter and Cluster work the same way with their own file and volume (ironflow-starter_postgres_starter_data, ironflow-cluster_postgres_cluster_data). In HA, .env pins the major with PGVERSION and SPILO_IMAGE: dump, start on fresh volumes with the new values, and restore.

Rotate the master key

IRONFLOW_MASTER_KEY encrypts every stored secret, and there is no rekey command. After you change the key, Ironflow cannot read the old values, so you set each secret again.

1. Before you change anything, list the secrets in each environment with ironflow secret list --env <env>. The command returns names only, so collect each value from its source.
2. Generate a new key with openssl rand -hex 32, and replace IRONFLOW_MASTER_KEY in .env.
3. Recreate the Ironflow containers: docker compose -f docker-compose.<size>.yml up -d. Until step 4 is done, functions that read a secret fail to resolve it.
4. Set every secret again with ironflow secret set <name> <value> --env <env>.

To undo the change before step 4, put the old key back in .env and run up -d again.

What Docker Compose does not do

Every size runs on one machine. Cluster survives a stopped Ironflow node or NATS server, and HA also survives a stopped PostgreSQL primary, but no size survives the loss of the host: if the machine goes down, the whole stack goes down. PostgreSQL is a single server in Single node, Starter and Cluster. For more than one host, run Ironflow against managed PostgreSQL and a NATS cluster (see the production checklist below).

Move to a larger size

The sizes use separate volumes, and NATS data does not move between them: JetStream streams cannot join a different cluster. The steps are the same for every move (Single node with PostgreSQL to Starter, Starter to Cluster, Cluster to HA, and so on).

1. Back up PostgreSQL from the current size with pg_dump.
2. Stop it: docker compose -f docker-compose.<size>.yml down (keep the volumes until the new size works).
3. Start the new size, and restore the dump into its PostgreSQL. Reuse your .env and add any keys the new size requires (.env.example lists them); for a new .env, run init-env.sh.
4. Set your secrets and config again (ironflow secret set …). They lived in the old NATS KV.

Production on your own infrastructure

For production across several hosts, run the engine container or the binary on each host (Docker, systemd, or another process manager) against managed infrastructure.

# Node 1
IRONFLOW_DATABASE_URL="postgres://ironflow:pass@db.example.com:5432/ironflow?sslmode=require" \
NATS_URL="nats://nats-1.example.com:4222,nats://nats-2.example.com:4222,nats://nats-3.example.com:4222" \
IRONFLOW_NODE_ID="node-1" \
IRONFLOW_MASTER_KEY="<openssl rand -hex 32>" \
IRONFLOW_JWT_SECRET="<openssl rand -hex 32, same on every node>" \
NATS_STREAM_REPLICAS=3 \
NATS_FILE_STORAGE=true \
IRONFLOW_STALE_CLAIM_THRESHOLD="2m" \
IRONFLOW_BLOB_URL="s3://my-bucket?endpoint=https://s3.example.com&region=us-east-1" \
./ironflow serve --bootstrap-key-file /data/bootstrap-key.json

# Node 2 (same config, different node-id)
IRONFLOW_NODE_ID="node-2" \
# ... same env vars ...
./ironflow serve --bootstrap-key-file /data/bootstrap-key.json

Production checklist:

• Managed PostgreSQL (RDS, Cloud SQL, etc.) with connection pooling
• 3-node NATS JetStream cluster with TLS and credentials
• NATS 2.14.7 or later on every server of an external cluster. ironflow serve refuses to start against an older server. Upgrade NATS before you upgrade Ironflow.
• NATS_FILE_STORAGE=true, so the five JetStream streams are on disk. Without it they stay in memory, and a restart of every NATS server empties them.
• Stable node IDs for log correlation
• Master key for secrets encryption, and the same JWT secret on every node
• A shared S3-compatible bucket for blobs (IRONFLOW_BLOB_URL, plus AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
• Load balancer in front of all nodes. Route on /health, not /ready: one stuck stream makes every node answer 503 on /ready, and routing on it takes every node out. Any balancing works, round-robin included: HTTP pull workers need no sticky sessions, because their registrations and job leases are in the database and any node serves any of their requests. When a node stops, workers carry on through the other nodes and keep the jobs they are running: a heartbeat on any node extends their leases. A job whose poll response, ack or result was lost with the stopped node runs again about two minutes later, once its 90-second lease expires and the engine queues it again. Make steps idempotent. Set a short connect timeout, as cluster/nginx.conf does (proxy_connect_timeout 2s), so a request moves to another node at once instead of waiting on the stopped one. If the nodes run in containers, have the load balancer resolve their names again while it runs, because a node that restarts can take a new address. cluster/nginx.conf does this through Docker’s DNS, so that file works only on the nodes’ Docker network. On another network, give it your own resolver, or remove resolve when the addresses never change.

Nodes use database-measured REST worker heartbeat age instead of comparing database timestamps with node time. When a read fails, or its timing bounds span a health threshold, polled cluster-health responses and pushed transitions retain that registration’s last published health. The reported heartbeat timestamp stays conservative. Its age can exceed the usual health threshold while classification is deferred. Local heartbeats and later reads replace the observation. Missing workers and workers definitively past the timeout still expire. Repeated uncertain reads can delay a stale worker’s local disconnect notice.

The cluster/docker-compose.multi-node.yml file is a smaller stack for developing against a cluster: three Ironflow nodes behind nginx, one NATS server, PostgreSQL and SeaweedFS. It runs the published image.

Requirements

Requirement	Single Node	Cluster mode (--nats-url)
Database	SQLite or PostgreSQL	PostgreSQL only
NATS	Embedded (automatic)	Bundled 3-node cluster or External NATS
Blob storage	Local filesystem for SQLite; explicit S3 for PostgreSQL	Shared s3:// backend (IRONFLOW_BLOB_URL)
Min nodes	1	1 (Starter) or more

Starting ironflow with --nats-url and SQLite will fail at startup with a clear error message. Starting it with --nats-url and no s3:// blob URL fails too: filesystem blobs are local to one node. See Blob overflow. The Starter, Cluster and HA files start a SeaweedFS service as that shared S3 backend and set IRONFLOW_BLOB_URL for you.

Architecture

Each Ironflow node is stateless. All coordination uses:

• NATS KV for cron deduplication (first-wins atomic claim)
• PostgreSQL SKIP LOCKED for distributed scheduler wakeups
• PostgreSQL dispatch_queue for job dispatch — workers long-poll GET /api/v1/workers/{id}/jobs (ADR 0037). The NATS STEPS stream is still created but no longer carries dispatch.

Configuration Reference

These flags and variables apply to ironflow serve without -f, which is how the compose files start it. With -f, the YAML file is the source of truth: set the matching field, or write ${VAR} in the file. IRONFLOW_BLOB_URL works with and without -f.

Flag	Env Var	Default	Description
--nats-url	NATS_URL	—	External NATS URL. nats://host:4222, tls://host:4222, or comma-separated seeds. Required for multi-node.
--nats-creds	NATS_CREDS_FILE	—	Path to NATS .creds file for JWT/NKey auth. See NATS credentials below.
--node-id	IRONFLOW_NODE_ID	random UUID	Stable node identifier. Recommended in production for log correlation.
—	IRONFLOW_DATABASE_URL	—	PostgreSQL DSN (postgres://user:pass@host:5432/db). Required for multi-node.
—	IRONFLOW_MASTER_KEY	(unencrypted)	Hex-encoded 32-byte AES-256 key for secrets at rest. Generate: openssl rand -hex 32. Omit only in local dev. Required by the Starter, Cluster and HA files.
—	IRONFLOW_JWT_SECRET	(required for PostgreSQL)	Hex-encoded 32-byte dashboard login signing key. Set the same value on every node, or a login made on one node fails on the others. Required by the Starter, Cluster and HA files.
—	IRONFLOW_BLOB_URL	—	Shared blob backend, s3://bucket?endpoint=...&region=.... Required for multi-node. Read with and without -f.
—	IRONFLOW_BOOTSTRAP_ADMIN_KEY	—	First admin API key (ifkey_ + 32 hex) stored at first boot. init-env.sh sets it.
—	IRONFLOW_STALE_CLAIM_THRESHOLD	2m	How long a scheduler claim can be held before another node reclaims it. Go duration string (e.g. 5m).
—	IRONFLOW_STALE_CLAIM_RECOVERY_INTERVAL	60s	How often the scheduler scans for orphaned claims. Go duration string.
—	NATS_STREAM_REPLICAS	1 (3 in docker-compose.cluster.yml and docker-compose.ha.yml)	Replica count for the JetStream streams and KV buckets. Set it to the NATS cluster size.
—	NATS_FILE_STORAGE	true	File storage for EVENTS, STEPS, RESULTS, PUBSUB, and CANCEL. false leaves them in memory, and a restart of every NATS server empties them. sync_interval does not apply to a memory stream. Outside these files the engine default is false.

The Starter, Cluster and HA files also require POSTGRES_PASSWORD, S3_ACCESS_KEY and S3_SECRET_KEY in .env, and HA also requires POSTGRES_SUPERUSER_PASSWORD and POSTGRES_REPLICATION_PASSWORD. SeaweedFS and Ironflow both read the S3 key pair from there.

NATS Credentials

If your NATS server requires authentication, pass a .creds file via --nats-creds. NATS credentials files are generated with the nsc CLI — Ironflow does not issue or manage NATS credentials.

# Example: generate creds with nsc (NATS operator tooling)
nsc add user --account MyAccount --name ironflow-node
nsc generate creds --account MyAccount --name ironflow-node > /etc/ironflow/node.creds

# Pass to Ironflow
ironflow serve --nats-url nats://nats:4222 --nats-creds /etc/ironflow/node.creds

How Coordination Works

Cron deduplication: When a cron expression fires, each node tries to Create a key in the SYS_cron_triggers NATS KV bucket. Only one node succeeds (atomic first-wins). The key expires after 24h. Duplicate runs are prevented without leader election.

Scheduler wakeups: When sleeping or waiting steps are ready, each node runs UPDATE ... FOR UPDATE SKIP LOCKED. PostgreSQL atomically assigns each row to one node. No step is processed twice. If a node crashes mid-processing, the stale claim recovery goroutine (runs every 60s) resets claimed rows after a configurable threshold (default: 2 minutes).

Execution recovery: Segments that crash while executing are not covered by the sweep above — they hold a concurrency lease, and the capacity scanner reclaims them when that lease expires, re-enqueueing each under a fresh execution_seq that fences any write from the crashed attempt. See Crash recovery.

Job dispatch: Pull-mode workers long-poll GET /api/v1/workers/{id}/jobs over HTTP. Admission is database-backed: a poll reserves a segment and mints a fenced lease, so any node can serve any worker. A running job survives a node crash: the worker’s heartbeat on any other node extends its leases (ADR 0138). Only a job whose poll response, ack or result was lost with the node goes through lease expiry: the capacity scanner re-enqueues its segment, and it runs again about two minutes later — see Crash recovery.

Monitoring

The Cluster and HA files (and the multi-node.yml development variant) ship a monitoring profile that brings up a full local observability stack alongside the nodes. Starter has no monitoring profile; the single-node file has a smaller one (Prometheus and Grafana).

docker compose -f docker-compose.cluster.yml --profile monitoring up -d

Service	URL	Purpose
Prometheus	http://localhost:9090	Scrapes /metrics from all three nodes
Jaeger	http://localhost:16686	OTLP trace backend (nodes export to jaeger:4317)
Grafana	http://localhost:3000	Dashboards, auto-provisioned

Prometheus evaluates the 20 engine and four NATS alert rules. The NATS exporter enables JetStream metrics. Host, PostgreSQL, probe, and notification alerts require the exporters described in deploy/docker-compose/monitoring/README.md and are omitted from Compose monitoring.

Each node sets IRONFLOW_METRICS_ENABLED=true and points IRONFLOW_OTEL_ENDPOINT at the bundled Jaeger by default. Grafana auto-provisions the Prometheus datasource and two dashboards (ironflow-performance, nats-monitoring) from deploy/docker-compose/cluster/grafana/.

Grafana allows anonymous read-only access; log in as admin (GRAFANA_ADMIN_PASSWORD, default admin) to edit. Set a real password before exposing the stack beyond localhost. Without --profile monitoring, nodes still expose /metrics but log harmless OTLP connection retries — set IRONFLOW_OTEL_ENDPOINT= empty to silence them.

For the metric catalogue, trace model, and Grafana queries, see the Observability guide. The Compose files ship no Alertmanager, so an alert shows in Prometheus and notifies nobody.

Known Limitations

• WebSocket subscriptions are node-local. Clients that reconnect to a different node lose live subscription state. NATS KV-backed subscriptions (Phase 6) will address this.

Production Considerations

NATS cluster quorum: A 3-node NATS cluster requires 2-of-3 nodes to write. Losing 2 nodes simultaneously will stall JetStream writes. Plan your cluster size and replication factor accordingly.

JetStream fsync: Every Compose file mounts a nats_conf config with sync_interval: always, so each NATS server fsyncs a write before it acknowledges it. nats-server has no command-line flag for this setting. Without it, nats-server acknowledges at once and fsyncs every two minutes, and a host crash or power loss can drop acknowledged secrets, config and KV writes. Fsync on every write lowers throughput; to trade durability for speed, change sync_interval to a duration such as 1s in the nats_conf block. See ADR 0134. Memory streams skip this fsync. The Starter, Cluster, HA, and multi-node files set NATS_FILE_STORAGE=true, so the five streams are on disk and the fsync covers them. KV buckets were already file-backed.

JetStream replication: Set NATS_STREAM_REPLICAS to match your cluster size so that one lost node does not lose data. Replication does not replace fsync: a power cut to every node can hit all replicas inside the same unsynced window. With -f, set spec.nats.streamReplicas instead. The default is 1 (no replication — fastest but not fault-tolerant). docker-compose.cluster.yml and docker-compose.ha.yml set 3. The count applies to the streams, to every system KV bucket (secrets, config, cron, debounce), and to KV buckets you create after startup. At startup each node sets the existing streams and system buckets to the count. KV buckets you created before keep their count; change one with nats stream edit KV_APP_<name> --replicas 3.

With more than one replica, a follower can answer a KV read. Right after a write on one node, a read on another node can return the previous value for a few milliseconds. Conditional writes (config If-Match, cron dedup, debounce timers) are checked by the stream leader and stay safe.

PostgreSQL connection pool: Each Ironflow node uses 25 max connections by default. With 3 nodes, plan for ~75 total connections.

PostgreSQL TCP keepalives: Every Docker Compose size runs PostgreSQL and Ironflow on one host, so it does not need them. When you run nodes on more than one host, set tcp_keepalives_idle, tcp_keepalives_interval and tcp_keepalives_count on the PostgreSQL server. Without them, a lost host keeps the migration lock for about 2 hours. See PostgreSQL across hosts.

Stale claim threshold: Tune IRONFLOW_STALE_CLAIM_THRESHOLD (default: 2m) based on your acceptable recovery latency after a node failure.

Graceful shutdown: Send SIGTERM to stop a node. A stop first gives dispatches in progress up to 30s, and a request that ends in that time counts as usual. A push request the node then cancels goes back to the queue with the same attempt, on its last attempt too, and runs again after the restart or on another node. The cancel does not use a retry. The endpoint can see the request again, as with any at-least-once delivery. If the node cannot write the requeue, or exits with a request still running, the run comes back through lease recovery instead, about 2 minutes later by default. When work is stuck, a stop can take about 90s: 30s for HTTP, 30s for dispatches, 25s for workers and 5s for follow-ups. Every Compose file sets stop_grace_period: 100s on its Ironflow services. If you use another orchestrator, set its stop timeout above 90s. A run waiting to resume at shutdown stays waiting and resumes after the restart.

Troubleshooting

external NATS requires PostgreSQL You set --nats-url but IRONFLOW_DATABASE_URL is not set. Multi-node requires PostgreSQL.

external NATS server version ... is unsupported The external NATS server is older than 2.14.7. Upgrade every NATS server in the cluster to 2.14.7 or later, then start ironflow serve again.

connect to external NATS: connection refused NATS is not reachable at the given URL. Check the URL, firewall rules, and NATS health (curl http://nats-host:8222/healthz).

timed out waiting for the schema migration advisory lock Another session held the migration lock for longer than IRONFLOW_MIGRATION_LOCK_WAIT (default 5m). The error names it: held by pid <N> (application_name ..., client_addr ..., xact_start ..., state ...). state idle in transaction is normal while a node migrates. The holder’s application_name (ironflow-migrate <node>) names its node, and xact_start shows how long it has held the lock. If that node is still running, it is still migrating: wait, or raise IRONFLOW_MIGRATION_LOCK_WAIT, and start this node again. Only if that node is no longer running, end the session, then start this node again:

SELECT pg_terminate_backend(<N>);

See PostgreSQL across hosts.

migration lock lost before <file> The node held the migration lock, then lost it, so it stopped and applied no more files. The transaction that holds the lock was killed (pg_terminate_backend, a database restart or failover), its connection broke, or the pooler closed it (a pooler restart, or PgBouncer’s idle_transaction_timeout). A file that was running when it happened finished, and may have run without the lock. Check that file and the one before it, and check for a second node that migrated at the same time. Then start the node again: it waits for the lock and applies what is missing. If a PgBouncer sits in front of PostgreSQL, set its idle_transaction_timeout to 0 or to more than the longest migration, because the holder is idle in a transaction for the whole run.

Steps stuck in waking status A node crashed after claiming steps. Wait for StaleClaimThreshold + 60s for automatic recovery, or run:

UPDATE steps SET status = pre_claim_status, claimed_by = NULL, claimed_at = NULL, pre_claim_status = NULL
WHERE status = 'waking' AND claimed_at < NOW() - INTERVAL '5 minutes';

Duplicate runs in database Cron deduplication requires the NATS KV SYS_cron_triggers bucket. Check NATS connectivity:

nats kv ls  # should show SYS_cron_triggers

/ready answers 503 and nginx reads unhealthy, while the API still serves A JetStream stream has a fault, and issues["stream:<name>"] in the /ready body names it. Accepted events wait in the outbox. A host reset can cause this on HA. See Recover a stuck JetStream stream.

After a host reset

A host reset or power loss in the first seconds after a new stack starts, or just after an image pull, can leave these errors. The kernel can hold a write in memory for about 35 seconds, and these components do not flush theirs. In our reset tests, Ironflow lost no write it had acknowledged. A stack that has run for a minute since its last pull did not show these errors. We saw each Docker case once or twice and have not measured how often they occur.

FATAL: no pg_hba.conf entry for host "...", user "ironflow", database "ironflow", no encryption Ironflow cannot log in to PostgreSQL, while PostgreSQL reports healthy. On Starter and Cluster the Ironflow container restarts in a loop. PostgreSQL lost the rule that allows TCP logins, and pg_isready does not log in, so the health check passes. The file ends in NUL bytes where the rule was. The Starter, Cluster, multi-node and single-node files now flush this rule during first start, so only a reset in a short window during PostgreSQL’s setup can cause this. We have not checked HA, where Patroni manages the file. To restore the rule and keep your data:

docker compose -f docker-compose.<size>.yml up -d postgres
docker compose -f docker-compose.<size>.yml exec postgres sh -c 'printf "\nhost all all all scram-sha-256\n" >> "$PGDATA/pg_hba.conf" && sync'
docker compose -f docker-compose.<size>.yml restart postgres
docker compose -f docker-compose.<size>.yml up -d

On a new, empty stack you can start again with docker compose -f docker-compose.<size>.yml down -v, then up -d. In the single-node file, add --profile postgres to down -v, or it keeps the database volume. down -v deletes the volumes and all their data, so it is safe only on a stack with no data.

panic: assertion failed: Page expected to be: 4, but self identifies as 0 dockerd does not start. The panic comes from bbolt. In the case we saw, the damaged file was the BuildKit cache: stop Docker, move /var/lib/docker/buildkit/cache.db aside, and start Docker. Moving this file cleared the error.

exec /bin/sh: exec format error on an image of the correct architecture A container from a newly pulled image does not start, because the image’s layers did not reach the disk. In the case we saw, only a new image store cleared this, and we have not tested a smaller step. On a new host with no data, the shortest route is a new host. Docker keeps named volumes in its data directory, so back them up before you change that directory on a host with data. To prevent this, run sync after each docker compose pull.

A stream has no leader: no leader after 30s, or context deadline exceeded on a stream A node exits at boot with stream EVENTS: no leader after 30s; its retention cannot be validated, or with a timeout such as failed to create stream PUBSUB: context deadline exceeded. The timeout means a stream had a leader when boot read it and did not take the write in time. A slow host can do this, and so can a stream losing its leader. Compose restarts the node, and the next boot reads the stream again. Check NATS:

docker compose -f docker-compose.<size>.yml logs --tail 100 | grep 'no leader for stream'

If this keeps printing catchup aborted, no leader for stream '$G > PUBSUB', each copy of the stream holds fewer messages than its own replication state promises. Each copy waits for a leader to send the rest, and none can become leader. Waiting and restarting NATS did not clear it, and we know of no NATS release that fixes it. We do not know the cause. We saw this once after a host reset, and later resets did not repeat it. We reproduced the same state by damaging the stream’s files on all three copies.

A PUBSUB, STEPS, RESULTS or CANCEL stream with no leader after 30 s does not stop a node. The node starts and reports the fault on /ready. See Recover a stuck stream before you delete anything. EVENTS and the system KV buckets still stop boot. Never delete a KV_ stream: those hold secrets, config, cron and debounce state.