What happens to a job when its worker shuts down?

Your deploy sends SIGTERM to an old worker. One job is halfway through its handler. Another assignment is already on its way. Killing the process would leave both jobs waiting for recovery, even though the second handler never started.

Ironflow’s Go and Node streaming workers can drain instead. They keep the connection open to report active work, refuse assignments they cannot start, and stop when the drain deadline arrives. A refusal returns work to the queue without spending a run attempt.

Let’s follow those two jobs through a rolling deploy. The replacement worker is connected and ready before the old worker receives its signal.

Read from top to bottom. A and B overlap; the timeline is not to scale. A’s completion and deadline recovery are alternative outcomes.

Job A is already running

The old worker accepted job A and started its handler. Five seconds into a 15-second operation, the deploy sends SIGTERM.

The worker enters draining. It stops accepting new jobs, but job A gets time to finish. Heartbeats and result messages still use the open stream. If A finishes within the deadline, the worker sends its result, flushes queued messages, and half-closes the stream. It waits for the engine to close its side before stopping, within that same deadline.

The run completes normally. Shutdown did not turn a successful handler into a failure or require a new attempt.

Here is a small Node worker with a deliberately slow operation. Use Ironflow and @ironflow/node v0.41.0 or newer, with Node.js 22 or newer. Save this as worker.ts in a TypeScript project. If you need a scratch project:

mkdir deploy-demo
cd deploy-demo
npm init -y
npm pkg set type=module
npm install @ironflow/node@0.41.0
npm install --save-dev tsx

import { setTimeout as delay } from "node:timers/promises";
import { createFunction } from "@ironflow/node";
import { createStreamingWorker } from "@ironflow/node/worker-streaming";

const processOrder = createFunction(
  {
    id: "deploy-demo",
    triggers: [{ event: "deploy.demo" }],
  },
  async ({ step, run, logger }) => {
    logger.info(`Starting run ${run.id}`);
    return step.run("process-order", async () => {
      await delay(15_000);
      return { processed: true };
    });
  },
);

const worker = createStreamingWorker({
  serverUrl: "http://localhost:9123",
  apiKey: process.env.IRONFLOW_API_KEY,
  functions: [processOrder],
  maxConcurrentJobs: 1,
  drainTimeout: 30_000,
});

await worker.start();

Run ironflow serve --dev in one terminal. This local engine does not require an API key. In the project’s terminal, start the worker:

npx tsx worker.ts

After registration, emit an event from another terminal:

ironflow emit deploy.demo --data '{}'

Press Ctrl+C in the worker terminal while the handler runs to exercise the SIGINT drain. A deploy’s SIGTERM uses the same drain path. The timer represents active handler work; step.sleep would yield the job back to the engine instead of keeping this worker busy. This demonstrates A’s drain. B’s refusal is an in-flight assignment race, not something every shutdown will produce.

The SDK already handles SIGINT and SIGTERM. It drains all running workers in the process, then exits. If your application installs its own signal listener, it owns process exit. Let the workers finish draining before exiting. You can also initiate a drain explicitly with await worker.drain(). worker.stop() stops immediately.

Job B arrives after the drain starts

Job B was offered just as the old worker entered its drain. That race matters: the assignment can already be in flight before the dispatcher knows this worker is draining. B’s handler has not started.

The worker sends JobNack with reason DRAINING, carrying the assignment’s execution sequence and lease token. The engine marks the session as draining, releases B’s lease and capacity slot, and moves its run from running to waiting. It queues a new execution sequence with no recovery grace period. The replacement worker can pick it up as capacity becomes available.

An assignment can put a run in running before the SDK starts its handler. Seeing B briefly enter that state does not mean B’s handler executed. The new sequence fences late messages from the refused assignment.

If the old worker were full rather than draining, it would send AT_CAPACITY. The engine would still requeue B, but hold back the worker’s overestimated capacity slot until a heartbeat shows space. It does not immediately send B back into that same full slot. This handles cases such as reconnects where the engine’s available credits exceed the worker’s actual free slots.

Refusal is different from failure

What happened	Retry attempt	Recovery count	What comes next
B never started; the engine accepts its JobNack	Unchanged	Unchanged	Runnable again without recovery grace
A’s handler reports a retryable failure	Uses the retry budget	Not a lease recovery	Retry if the function’s policy allows it
A disappears without a terminal result	Not a handler-failure retry	Counts toward the recovery ceiling	Lease expiry, recovery grace, then redispatch

A retryable JobFailed is the wrong way to refuse work. It spends the run’s retry budget and can fail a run on its last allowed attempt. JobNack says “I did not start this assignment,” so it changes neither attempt nor recovery_count and writes no handler error.

If A outlives the deadline

The Go and Node SDK drain default is 30 seconds. An engine Shutdown message can supply its own deadline; the engine’s default is 25 seconds. The worker uses a positive timeout from that message instead of its configured timeout. A process signal starts the SDK’s configured drain. In Node, a drain already in progress keeps its original deadline.

Give the process supervisor enough termination grace for the drain and exit. SIGKILL bypasses the SDK entirely. At the drain deadline, the worker cancels remaining jobs and closes the stream. Cancellation cannot undo an external write, and arbitrary handler code may keep running until the process exits.

If A has no recorded terminal result, the engine recovers it through lease expiry. Current defaults are a 90-second lease, a 30-second scanner interval, and a 30-second recovery grace period. The lease clock starts at its last refresh, not at SIGTERM. Scanner and dispatch scheduling add time too. With a healthy engine and available replacement capacity, the ordinary crash-recovery path takes roughly 90 seconds to three minutes from loss of the worker to resumed execution, before the job’s own runtime. Making B runnable immediately after a nack is a different path.

Lease recovery counts toward RecoveryCeiling, which permits 10 consecutive recoveries by default. The next recovery fails the run with EXECUTION_RECOVERY_EXHAUSTED. An accepted yield that releases the lease resets this count. Completing a step inside a still-running job does not. A refused job does not spend that allowance.

Completed durable steps whose results reached the engine can be reused. An external write that succeeded before its step result was recorded may happen again. Give that destination an idempotency key or a uniqueness constraint. Graceful drain and execution fencing do not make external side effects exactly once.

What to watch during the deploy

The Node worker logs Draining worker... and, while A runs, Waiting for 1 jobs to complete.... An in-flight B produces Draining, refusing job. A capacity refusal produces At capacity, refusing job. For an accepted nack, the engine logs capacity: job nacked, re-queued with no attempt used, with the run ID and reason. The run’s attempt stays the same as it moves through waiting and back to running on the replacement worker.

If A exceeds the deadline, look for Drain deadline reached; cancelling 1 active jobs. Lease recovery emits the audit events capacity.lease.expired and capacity.segment.recovered. After recovery, waiting can include the recovery grace period; it is not proof that the replacement worker is broken. The crash recovery explanation shows how to inspect the leases and the queued segment’s eligibility time.

If a refusal cannot reach the engine, immediate requeue is unavailable. The streaming heartbeat refreshes leases only for reported jobs, with a compatibility fallback for older workers that omit the job list. An unreported job can therefore recover after lease expiry even while a worker stays connected. That fallback recovery does count toward the recovery ceiling.

Roll out the engine before the workers

v0.40.0 added streaming step.invoke support. v0.41.0 added graceful drain and JobNack. Upgrade the engine first, then the streaming SDKs. An older engine ignores the new nack message, so a new worker’s refusal does not get immediate requeue. A newer engine with an older SDK can use the heartbeat backstop, subject to the compatibility fallback. Both sides need nack support for the refusal path described here.

When upgrading from pre-v0.40.0 streaming SDKs, update every participating Go and Node worker. Those older SDKs can interpret a failed invoke as success with empty output. See the Node streaming worker reference for protocol behavior and rollout notes, and the API protocol reference for the WorkerService transport. Keep a ready replacement, allow the old worker to drain, and distinguish a refused assignment from a handler failure when reading the runs.