Recover a Stuck JetStream Stream

/ready answers 503, or the JetStreamStreamFault alert fired, and the API still serves requests. A JetStream stream cannot do its work. This page shows you how to read the fault, what works while it lasts, and what you can do about it.

No recovery that keeps a stuck stream is known yet. The only known way out is to delete the stream, and that has a real cost. Read Delete the stream before you run it.

Recognize the fault

Ask /ready. It needs no credentials. Use port 9000 on Cluster and HA, and 9123 on Starter.

curl -s http://localhost:9000/ready | jq

{
  "status": "not ready",
  "issues": {
    "stream:PUBSUB": "no leader"
  }
}

The key is stream: plus one of EVENTS, STEPS, RESULTS, PUBSUB or CANCEL, or stream:KV_ plus a system bucket’s name, such as stream:KV_SYS_debounce_state. The value is one of five fixed texts:

Text	What the node read	Reported after
no leader	A member of the stream answered and named no leader. A member answers this way about 10 s after the fault starts, so /ready turns red about 45 s after it.	30 s from the first such answer
no answer	NATS lists the stream, and no member answered. Every stream reads this when the node has had no good read for 60 s, as during a NATS outage.	60 s
missing	NATS does not list the stream.	30 s
replaced	Two leader replies in a row carry a new creation time, so the stream was deleted and created again.	About 5 s
config pending	The node could not write its stream config at boot. It reads this after a leader returns, until it has written the config and started the stored consumer groups. If NATS does not answer a consumer create within 30 s, the node tries again after its next read, while the stream keeps its leader, and logs each attempt. After 5 failed attempts, about 3 minutes, it logs the groups it could not start at error level and the stream reads green; each of those groups starts on its next join. It also reads this in the first 30 s after a boot that got no answer.	See the row text

A system bucket reads only no leader or no answer. A bucket that is deleted leaves the list, and a bucket that is made again is not replaced.

You can see the same fault in four other places:

• Metrics. ironflow_jetstream_stream_fault{stream,fault} is 1 while /ready reports that fault. ironflow_jetstream_stream_leader_missing_seconds{stream} counts how long the stream has had no leader, no answer or no listing, and is 0 otherwise. The bundled JetStreamStreamFault alert fires when the fault gauge holds for 1 minute.
• Logs. An error line repeats every 30 s with the stream field and a message such as stream PUBSUB no leader for 45s. A boot that left a stream alone logs Stream has no leader; its config stays pending until a leader returns.
• docker ps. On Cluster and HA, nginx reads unhealthy when no node answers /ready through it. The Ironflow containers stay healthy, because their healthchecks read /health. nginx keeps sending requests to every node, so the API still serves. NATS servers that hold a stuck copy read unhealthy too, and docker compose up logs optional dependency "nats-2" failed to start and starts the Ironflow containers anyway, after about 90 s. docker compose up --wait fails, because nginx reads unhealthy.
• NATS. nats --server nats://127.0.0.1:4222 stream info PUBSUB shows the stream’s cluster state. Compose does not include the nats command. If your host has none, run it from a container: docker run --rm --network host natsio/nats-box:latest nats --server nats://127.0.0.1:4222 stream info PUBSUB. In a stuck stream, the reply names no leader and can show 0 messages.

A short fault is normal. Members stay silent during an election, and a NATS server waits about 10 s after it starts before it reports a group with no leader. If a leader returns, the node logs Stream leader is back and Stream config applied after its leader returned, and /ready answers 200. You do nothing.

Why a stream stays stuck

On Compose HA, a host reset can leave a stream with 3 replicas with no leader for good. The fault starts in NATS (nats-server 2.15.2). From its source, each replica pauses to catch up from a leader, and a paused replica cannot stand for election, so none becomes leader.

We saw this once after hcloud server reset on HA on cx23, and we do not know the cause. 93 later resets of that setup did not repeat it, which is at most about 3.2% per reset at 95% confidence, if each reset is independent. We reproduced the same state by cutting short the message files of all three copies after a raft snapshot. Damage to 2 of 3 copies recovered by itself.

To confirm it, read the NATS logs. A stuck stream keeps printing catchup aborted, no leader for stream '$G > PUBSUB':

docker compose -f docker-compose.<size>.yml logs --tail 100 | grep 'no leader for stream'

What still works

The node keeps serving while the fault lasts.

• /health answers 200, and API reads answer from the database.
• Emit and entity stream appends are accepted. Each one commits to the database with an outbox row.
• Accepted events wait in the outbox and normally do not dead-letter. See the outbox for the exceptions. The outbox worker publishes them once NATS takes them. Watch ironflow_outbox_pending_count and ironflow_outbox_oldest_pending_age_seconds. The outbox explanation has the details. If the fault outlasts the 2-minute dedup window, a publish that reached NATS and lost its ack can publish twice, so a SQL projection handler must be idempotent.
• Functions triggered by an accepted event get their runs. Emit creates the runs in the database and dispatches them.
• KV, secrets and config are unaffected by a PUBSUB fault. Each KV bucket has its own stream.

What is broken while PUBSUB has a fault

Feature	What you see
Topic publish	PubSubService/Publish answers unavailable once its publish retries fail, so a client retries. It refuses nothing before the publish, so it is not on the 503 list below. It can keep answering unavailable for up to about 30 s after /ready answers 200, because the publish breaker that opened during the fault is still open. It clears on its own; keep retrying.
Wildcard subscribe	The subscribe needs a JetStream consumer on PUBSUB. It fails and the subscription ends, so the client retries. A subscribe that resumes from a sequence fails the same way. A replay that fails is logged, and the subscribe goes on without history.
Consumer groups	Create, update, delete and join are refused. Members that already joined get no messages, because no new message reaches the stream. A node that booted during the fault leaves its stored groups idle until a leader returns.
Projection delivery	SDK projection pages answer unavailable, and the runners retry. SQL projections hold. Read models go stale: queries still answer, from old data.
Catch-up waits	WaitForEvent waits for the outbox to publish the event, and answers deadline_exceeded at your timeout. Other catch-up waits time out because the read model does not advance.
Rebuilds	A new rebuild is refused. A rebuild that started earlier fails with an outbox error if an event from before its start stays in the outbox for 15 minutes.
System frames	system.* frames are not in the outbox. A frame that fails to publish is lost for good. The dashboard’s live updates use these frames and wildcard subscriptions, so they stop. Each lost frame leaves a row with source publish in ironflow dead-letters list. The row has no payload, so you cannot replay it.

An exact-topic subscribe with no replay and no resume sequence attaches over core NATS, without the stream. Its attach does not show that the stream is healthy.

What answers 503

While PUBSUB has a fault, these doors refuse before any write, so a refused request changes nothing. The message is the stream name and the fixed text, such as stream PUBSUB: no leader.

Door	Answer
POST /api/v1/environments, DELETE /api/v1/environments/{id}	503
POST /api/v1/tenants/provision	503
DELETE /api/v1/projects/{id}	503
DELETE /api/v1/orgs/{id}	503
DELETE /api/v1/platform/tenants/{id}	503
EnvironmentService/CreateEnvironment, DeleteEnvironment	unavailable
ProjectionService/RebuildProjection, except a dry run	unavailable
ProjectionService/CreateSQLProjection	unavailable
PubSubService/CreateConsumerGroup, UpdateConsumerGroup, DeleteConsumerGroup	unavailable
PubSubService/JoinConsumerGroup, Subscribe with a consumer group, SubscribeBidirectional joins	unavailable
WebSocket subscribe with a consumer group	unavailable in the subscription_result

The CLI and the SDKs reach the same doors, so they get the same answers. Retry these requests after recovery.

These doors read PUBSUB only. A fault on another stream makes /ready answer 503 and leaves them open.

Recover

1. Read the fault. Use the table above. On missing or replaced, each node restarts itself unless the stream is EVENTS: it exits non-zero after a random delay of up to 60 s, your supervisor starts it again, and boot rebuilds the stream’s consumers. For PUBSUB, then follow Bring projections back. For EVENTS, see EVENTS and system KV buckets.
2. Wait for a leader. An election clears on its own. For a stream that stays stuck, these did not clear the measured fault: waiting more than 20 minutes, stopping the engine, restarting one or all NATS servers, and running 2.15.3-RC.1.
3. Decide whether to stay degraded. Accepted events are safe in the outbox while you wait. The cost is stale read models, no topic publish and lost system frames. If you can carry that, keep the stream and wait for a fix.
4. Delete the stream only as a last resort.

Delete the stream

This replaces the stream

nats stream rm PUBSUB clears the stuck state, but it is not a clean recovery. It discards every message the stream holds and gives every projection and consumer group a stream it has never seen. You cannot undo it.

Starter, Cluster and HA publish the first NATS server on 127.0.0.1:4222.

nats --server nats://127.0.0.1:4222 stream rm --force PUBSUB

Do not create the stream yourself. Each node reports missing after 30 s and exits non-zero after a random delay of up to 60 s, so the nodes do not all restart at once. Compose restarts them, and the first boot creates PUBSUB. A node that is still running then reports replaced and exits the same way, so the last node can exit about 2 minutes after the delete. Requests to a node that is restarting can fail, so retry them. Wait until /ready answers 200.

The replacement has these consequences:

• SDK projections are refused. Their reads answer failed_precondition, and a rebuild is refused the same way.
• SQL projections go to status error within 30 seconds. Nothing more applies, and a rebuild is refused.
• Consumer group durables are rebuilt as DeliverNew. Boot starts each stored group at the next new message. Unread messages are gone, and no action brings them back.
• Topic history is gone. A replay finds only what was published after the replacement.
• A repeated idempotency key publishes nothing. The server keeps these keys in its database, not in the stream, so the delete does not clear them. A publish with a key used before the delete returns duplicate with the first event_id, and the new stream does not get the message. To publish it again, use a new key. See how long the server remembers a key.

Events waiting in the outbox publish into the new stream.

Bring projections back

List them, then recover each one:

ironflow projection list

For an SDK projection, delete it and restart its runner, which registers it again:

ironflow projection delete <name>

For a SQL projection, keep its definition at hand, because the delete removes it. Then delete it and create it again:

ironflow projection delete <name>
ironflow projection create <name> --sql-file <ddl.sql> --event <event> --event-handler '<event>=<SQL>'

For an external projection, clear your own store as well. See ironflow projection rebuild.

The new read model holds what the new stream holds: events published after the replacement. Older events stay in the events table, but no projection receives them. A rebuild does not bring them back: it answers failed_precondition while that table holds the older stream’s events. No supported path puts them into a read model today.

EVENTS and system KV buckets

The steps above are for PUBSUB. EVENTS and the system KV buckets do not follow them.

• A stuck EVENTS stream refuses boot. A node exits with stream EVENTS: no leader after 30s; its retention cannot be validated, because the engine must validate its retention before anything consumes it. A node that is already running stays up and reports the fault, so do not restart it.
• A system KV bucket with no leader is reported, and the node starts. /ready names it as stream:KV_SYS_<bucket>, with the environment ID of a secrets or config bucket replaced by <environment>, the gauges and the alert carry the full name, and boot binds the bucket without writing to it. Stopping two NATS servers can take a bucket with PUBSUB, because each bucket has its own replicas. While a bucket has no leader, a read of one key answers from the remaining copy, which can be behind the last write, and writes fail and say so. An emit for a function with a debounce window is accepted, and the engine arms it by itself once the bucket answers again, within about three minutes (SYS_debounce_state); an event that arrives over NATS is redelivered until it can be armed. ironflow secret set, ironflow secret delete and a config write answer 503 unavailable with retryable: true, so retry them after recovery (SYS_secrets_<env>, SYS_config_<env>). ironflow secret list and GET /api/v1/config answered with data with two of a bucket’s three replicas stopped on Compose HA. Do not rely on that: a listing asks the stream for state, and it failed the same way on an in-process test cluster. An agent tool registration fails (SYS_agent_tools). A secret read that gets no answer fails its run with SECRETS_UNAVAILABLE, and that run is not retried. Everything recovers on its own when a leader returns, with no restart. A debounce bucket that answers nothing at all when the engine opens it still refuses boot, with the bucket’s name in the error. Do not delete a secrets or config bucket: its contents have no other copy.
• A missing or replaced EVENTS only reports. The node stays up, /ready stays red, and the log says to follow the stopped-writer procedure. Do not run stream rm EVENTS as a recovery, and do not restart the nodes first. Each new EVENTS stream takes one of a fixed number of permanent fences, and a boot refuses a new stream that already holds messages. Follow Upgrade an existing EVENTS stream, and see Accounting limits and recovery.

Route on /health

Point a load balancer at /health. /ready means that the node can do everything, so one stuck stream makes every node answer 503. A load balancer that routes on /ready then takes every node out, and a partial fault becomes a full outage. See GET /ready.