Ironflow v0.55.0

v0.55.0 makes projections safe across a recreated stream, keeps accepted events through a NATS outage, and adds end-to-end W3C tracing, ironflow events explain and a dead-letter list. It removes the Helm chart and ironflow deploy, tightens the Compose files, and changes SDK timestamps to Date and time.Time. Read Breaking Changes before you upgrade. Stop every node of the older version first.

Breaking Changes

Upgrade and deployment

• The Helm chart, ironflow deploy, ironflow deploy vps, ironflow provision and engine HCLOUD_TOKEN are removed (#3588). Run the engine container (ghcr.io/sahina/ironflow-releases) with the Docker Compose files. The oci://ghcr.io/sahina/charts/ironflow chart is no longer published. The engine no longer reads KUBECONFIG. Alert rules are plain files under deploy/docker-compose/monitoring/ (#3599).
• docker-compose.multi-node.yml publishes only nginx on all interfaces (#3777). NATS, PostgreSQL, SeaweedFS and the node API ports bind to 127.0.0.1. Use an SSH tunnel to reach them from another machine. examples/financial-rag does the same for PostgreSQL (#3778).
• The Compose files serve /metrics on a separate port (#3764). Each node serves it on its API port plus 100 (9223, 9224, 9225), on 127.0.0.1 only. The API port answers 404. Prometheus and Grafana bind to 127.0.0.1, and so do Jaeger and its OTLP port in the multi-node file. Replace the Compose files and the image together. The new files need ironflow serve --metrics-port.
• Compose cluster Prometheus uses the ironflow scrape job, not ironflow-cluster (#3599). The 37 former Helm alerts are plain appliance rules.
• Compose cluster stacks keep the five JetStream streams on disk (#3675). A stack whose streams are still in memory does not start until you delete them. Deleting them discards their messages. Do not delete the NATS volume.
• PostgreSQL serve requires IRONFLOW_BLOB_URL (an s3:// URL), IRONFLOW_JWT_SECRET and --bootstrap-key-file. YAML uses spec.blobs.url, spec.auth.jwtSecret and spec.auth.bootstrapKeyFile. SQLite defaults do not change. docker-compose.single-node.yml writes the key to /data/.ironflow/.ironflow_bootstrap_key.json.
• IRONFLOW_CLUSTER_ID is a database identity assertion. The first start saves the cluster and lifetime IDs. A later mismatch refuses to start.
• Dashboard JWT key rotation needs persistent key storage. Set spec.auth.jwtSecretFile, --jwt-secret-file or IRONFLOW_JWT_SECRET_FILE. An inline secret still starts the server, but rotation returns HTTP 409.
• /ready answers 503 while a replicated JetStream stream has no leader (#3793). The engine exits when one of its streams is deleted or replaced while it runs. The Compose node healthchecks read /health, and nginx keeps /ready.
• EVENTS requires positive JetStream acceptance. Publish to the ingest and resume subjects and wait for the PubAck. A full stream rejects new messages and does not evict accepted ones. The first registration of an EVENTS incarnation needs an empty stream, and consumption needs $SYS.REQ.USER.INFO permission. A rollback to an older version is not allowed.
• Consumer group durables are named per group incarnation (#3827). Durables are named group-<environmentID>--<groupID>. After the upgrade, each group starts at the live tail at its first activation. Let workers drain first. The old cg-… durables stay on PUBSUB and count in ironflow_nats_consumer_lag. Remove them with nats consumer rm.
• Projection durables are named per projection incarnation (ADR 0163). A projection created under the name of a deleted one starts from the beginning. Remove old durables with ironflow projection durables prune --delete. Projection names are limited to 128 characters.
• Migrations 107 to 113 run on upgrade. On PostgreSQL, migration 108 rewrites outbox under a table lock, so emits wait. Do not requeue outbox dead letters until every node runs this release.

Projections

• SDK projection runners must send a delivery position on every poll and stream (ADR 0164, #3629). A request without one gets failed_precondition. last_event_seq in the poll response is now the delivery watermark. New runners stop with SERVER_TOO_OLD against a server older than 0.55.0.
• External projections refuse a partition filter and a partition rebuild (ADR 0164).
• A projection is bound to the PUBSUB stream it was registered against (ADR 0164). After the stream is recreated, an SDK projection gets failed_precondition and a SQL projection stops with status error. A projection from an older release has no binding: delete it and register it again. A rebuild is also refused when the events table holds another stream’s events, or when the stream no longer holds the last event the runner committed.
• A failed rebuild refuses every later read until a rebuild succeeds (#3716). GetProjection on a managed projection and catch-up waits answer failed_precondition. A replay skips a projection update from a failed rebuild. A SQL projection whose rebuild failed or was cancelled stops until a new rebuild.
• A projection registration that changes the event filter, partition key or mode during a rebuild is failed_precondition (#3661).
• Events ingested on events:{name} carry the raw payload (#3626). The {"event","data","timestamp"} envelope is gone. Rebuild affected projections after the 7-day PUBSUB window passes.

SDKs and API

• Go timestamps are time.Time (ADR 0165). This covers runs, steps, projections, tenancy, access, secrets, event schemas, audit, KV, config, functions, workers, streams and webhooks. Optional fields are *time.Time. A missing required timestamp fails the call with INVALID_RESPONSE.
• @ironflow/core, @ironflow/node and @ironflow/browser timestamps are Date values (ADR 0165). An unreadable value throws SchemaValidationError. Use timestampExact for entity history bounds and updatedAtExact for expectedUpdatedAt on webhooks. The ClientRun, ClientListRunsOptions and ClientListRunsResult aliases are removed.
• GET /api/v1/tenants returns created_at as a full RFC 3339 UTC timestamp. It was a date. The Go SDK fails against an older engine.
• streams.read() with no limit returns every event (#3851). ReadStream with limit unset or 0 used to return the first 100. The MCP tool ironflow_read_entity_stream keeps its 100-event bound.
• DEFAULT_RETRY.MAX_ATTEMPTS in @ironflow/core is 25, the server default (#3851). It was 3.
• Browser subscribe calls own independent server subscriptions (#3620). Duplicate patterns no longer reject, and each one uses a subscription slot.
• ironflow platform subcommands and ironflow policy --platform refuse --as-org (#3851).
• ironflow mcp --transport streamable-http refuses a request body over 4 MiB with 413.
• A run.created audit row’s created_at is the run’s creation time (#3735), and audit event IDs are opaque strings (#3725). A run.created ID is run_created_<runID>. Sort by created_at, overlap your polls, and drop duplicates by ID.

Observability and policy

• Log identity fields are unified (#3431). Use run_id, function_id, step_id, environment_id and environment_name.
• Metric label names are unified (#3421, #3422). Use environment_id, function_id and projection. Update external queries and reimport the shipped dashboards.
• Authorization IRNs escape a literal % as %25 (#3452). A stale deny rule for a name with % can stop matching. Review your policies.
• HTTP audit decision paths store the route pattern or unmatched (#3346, ADR 0148).
• The docs sites need Chromium 116, Safari 17.4 or Firefox 125 or later.

Features

• Explain event routing without emitting an event. Use ExplainEventMatch, ironflow events explain, the read-only MCP tool ironflow_explain_event_match, or the dashboard’s expanded event rows. Each visible active function gets a bounded reason.
• List dead letters (#3427). Use GET /api/v1/dead-letters, ironflow dead-letters list, or MCP ironflow_dead_letter_list. It needs an explicit environment and events:subscribe.
• W3C trace context runs through the engine and the SDKs (ADR 0152, #3434–#3440). The trace survives NATS, the outbox, push segments and pull assignment. Go, JavaScript and Python workers add process and executed-step spans. Python adds the optional tracing extra.
• Correlated SDK logging (#3444, #3445, #3446). JavaScript userLogger, Python CorrelationFormatter and Go Context.Logger() receive the run, function, step, attempt and trace IDs. createOTelLogger is an opt-in bridge.
• Instance freeze and thaw. Platform-only REST and Connect controls freeze new execution while active results, heartbeats and cancellation keep working. Freeze, restart and thaw keep pending work.
• A coding agent can set up Ironflow from one sentence. Tell it: Read https://ironflow.run/start.md and set up Ironflow in this project. ironflow skills sync --agent claude|codex|gemini and ironflow init my-app --agent claude wire the skills.
• Temporary log level change (#3432). POST /api/v1/cluster/log-level is audited. The override expires after 15 minutes by default.
• More runtime metrics. Debounce outcomes and latency (#579), Go and process metrics, connection pool gauges, push attempt outcomes (#3430), and sampled outbox, NATS lag, dead-letter and rebuild gauges (#3429). A scrape no longer queries the database.
• Run detail keeps the matched cancelOn rule and cancelling event. REST and Connect expose them. The push circuit breaker keeps its last failure reason, and run responses report claimed_by (#3426, #3433).
• serve in @ironflow/node accepts form-encoded webhooks, the Twilio default (#3851).
• ironflow subscribe, projection watch, stream subscribe and inspect send IRONFLOW_API_KEY, --as-org and IRONFLOW_ENV (#3851).

Fixes

• A node that missed a policy invalidate no longer keeps allowing what an edit revoked (#3826). Each node reads the store on every refresh and answers the new epoch within 30 s.
• A NATS outage or a leaderless stream no longer moves accepted events to the outbox dead letters (#3793). The outbox holds the row and publishes it when NATS recovers. External NATS connections retry without a limit.
• A system KV bucket with no leader no longer crash-loops boot (#3813, #3815, #3793). /ready names the bucket or stream. A boot that times out on a stream now says what that can mean (#3765).
• A clean shutdown no longer strands or fails runs (#3659, #3670, #3687). A cut-off dispatch requeues with the same attempt. Compose files set stop_grace_period: 100s.
• PostgreSQL deadlocks in stream purge, environment delete, run pruning, RedactEvent and step reports are fixed (#3740, #3760, #3761, #3788, #3794, #3795, #3822). A purge returns aborted and deletes nothing when another call holds a run. A purge no longer leaves capacity held (#3823).
• A delete that loses a race returns 409 aborted, not 500 (#3831). The CLI exits 7 for it. A project read failure is no longer 404.
• A stuck or locked table no longer stalls other work (#3658, #3660, #3686, #3721, #3738, #3743). Sleep wakeups, parent wakes, the waiting-run pass and freeze or thaw wait at most 1 s for a lock, or 500 ms for freeze.
• A rebuild no longer skips events or wipes state (#3628, #3644, #3662, #3672, #3679). It waits on a pending outbox row, fails when retention deleted an unread event, and checks its refusals inside the start transaction.
• A run cancelled by cancel_on publishes its run event, audit row and CANCEL message. A run failed at the recovery ceiling sends its RESULTS frame. RESULTS and CANCEL subjects name the run’s project, so TriggerSync works outside default (#3657).
• A desktop session minted from an API key can no longer change that key’s roles or rotate it (#3811). Platform key audit rows no longer land in an impersonated tenant (#3810). Audit rows now name dashboard and platform callers (#3043).
• A late report from an expired streaming worker can no longer complete the recovered run (#3043).
• An event redacted before it is published now publishes the placeholder (#3683). Run RedactEvent again for events redacted during an outage (#3809).
• A Files API read that the blob backend cannot serve ends with 503 BLOB_BACKEND_UNAVAILABLE (#3705). A host reset no longer loses acknowledged files (#3673).
• A change to a function’s recording setting applies without a restart (#3759).
• An event published to events.ingest.> reaches PUBSUB even when the first publish is rejected (#3626). An envelope with no data persists as JSON null (#3455).
• Compose healthchecks are correct (#3779, #3765, #3726, #3715). Multi-node reports healthy nodes as healthy. nginx re-resolves node names and reads unhealthy when no node is ready. PostgreSQL setup survives a host reset.
• A frozen instance no longer logs refused work as errors (#3791).
• Go SDK timed and parallel callbacks propagate panics to their caller (#3515).
• CreateConsumerGroup returns the real createdAt and updatedAt. A consumer group frame with includeMetadata carries meta.timestamp and the sequence (#3697).
• The Go, Node and Python projection runners stop on failed_precondition and SERVER_TOO_OLD, not reconnect forever (ADR 0164).
• Rolling a function back clears the circuit breaker its old endpoint opened. ironflow_projection_rebuild_duration_seconds records rebuilds again. A SQL projection that cannot drop its table now fails and stays in place.
• CLI and help fixes (#3851). ironflow server info --server prints that URL, the go-quickstart projection counts past 1, and a WebSocket replay over the limit answers invalid_argument.

Improvements

• Event ingest and resume processing are bounded and fair (#3587, #3684). Durable broker receipts and freeze admission cost some throughput: about 2% on API-triggered runs and about 24% on NATS-ingested runs in a two-node test.
• Entity event retention keeps an event until its triggers start (#3587). A step.invoke child honors a cancel_on event stored on another node or before a restart (#3709).
• Audit writes use at most a quarter of the PostgreSQL pool and have a 5 s limit (#3656, #3043). A dropped write counts in ironflow_audit_write_failures_total. Alert on that metric.
• ironflow mcp over stdio answers every tools/call it read before stdin closed (#3741). The MCP library moves to 1.5.0.
• Dashboard errors show the server’s message (#3043), and Compliance is now Entity Audit (#3664).
• Docs. The docs home is shorter and no longer describes a source build. Use-cases show twelve cases with a flow diagram each. The compatibility policy for 1.x is documented and takes effect at 1.0.0 (#3006).