Error Handling

Ironflow provides automatic retries for transient failures while allowing you to mark permanent failures that shouldn’t be retried.

Retry Behavior

By default, errors in step functions are retried according to your function’s retry configuration. When you set nothing, the server’s defaults apply:

Setting	Default	Meaning
maxAttempts	25	Total attempts, including the first (so 24 retries)
initialDelay	1s	Base of the backoff
backoffFactor	2.0	Exponential backoff multiplier
maxDelay	5m	Longest wait between attempts
onExhausted	pause	What happens when the attempts run out: pause or fail

At those defaults a failing run keeps trying for about 88 minutes. The waits are 2s, 4s, 8s … 256s, then 5 minutes each. An outage of an hour or so is ridden out without anyone being paged.

You can change the defaults for the whole server with spec.engine.retry in the YAML configuration, or set a policy on one function. A function’s own setting wins, field by field, and any field you leave out keeps the server default.

createFunction(
  {
    id: "my-function",
    triggers: [{ event: "my.event" }],
    retry: {
      maxAttempts: 5,        // Total attempts, including the first (so 4 retries)
      initialDelayMs: 1000,  // Initial delay (1 second)
      backoffFactor: 2.0,    // Exponential backoff multiplier
      maxDelayMs: 300_000,   // Maximum delay (5 minutes)
      onExhausted: "fail",   // "pause" (default) or "fail"
    },
  },
  async ({ event, step }) => { /* ... */ },
);

@function(
    id="my-function",
    triggers=[{"event": "my.event"}],
    retry={
        "max_attempts": 5,      # Total attempts, including the first (so 4 retries)
        "initial_delay": 1,     # Initial delay (1 second; bare numbers are seconds)
        "backoff_factor": 2.0,  # Exponential backoff multiplier
        "max_delay": "5m",      # Maximum delay (5 minutes)
        "on_exhausted": "fail", # "pause" (default) or "fail"
    },
)
async def my_function(ctx): ...

ironflow.CreateFunction(ironflow.FunctionConfig{
    ID:       "my-function",
    Triggers: []ironflow.Trigger{{Event: "my.event"}},
    Retry: &ironflow.RetryConfig{
        MaxAttempts:   5,               // Total attempts, including the first (so 4 retries)
        InitialDelay:  time.Second,     // Initial delay (1 second)
        BackoffFactor: 2.0,             // Exponential backoff multiplier
        MaxDelay:      5 * time.Minute, // Maximum delay (5 minutes)
        OnExhausted:   "fail",          // "pause" (default) or "fail"
    },
}, handler)

Retry delays follow exponential backoff. With the settings above:

1. First retry (after attempt 1 fails): about 2 seconds
2. Second retry (after attempt 2 fails): about 4 seconds
3. Attempt 5 is the last — maxAttempts counts the first attempt, so 5 means 4 retries

(The engine computes initialDelayMs × backoffFactor^(n-1), where n is the number of the attempt that is about to start. n is 2 for the first retry. The result is capped at maxDelayMs, then ±10% jitter is added — so the delays land near 2s and 4s, not exactly on them.)

The attempts and the exhaustion policy are fixed on each run when it is created. Changing a function’s policy, or the server default, affects the runs created afterwards and never a run that is already retrying.

When retries run out

A retryable failure on the last attempt follows the run’s onExhausted policy:

• pause (the default). The run becomes paused with pause reason retries_exhausted and keeps its last error. It releases its capacity, keeps every completed step, and waits. An operator fixes the cause and runs ironflow run resume <run-id>, or resumes it from the dashboard, which grants it a fresh budget and continues from the last completed step. The run emits system.run.{id}.paused and counts in ironflow_runs_paused_exhausted_total, so you can alert on it. A parent run waiting on it with step.invoke keeps waiting, and a synchronous caller (TriggerSync, InvokeFunctionSync) returns at once with status PAUSED and the last error.
• fail. The run becomes failed, as it always did. Saga compensations run, system.run.{id}.failed is published, and a waiting parent is woken with the failure.

A non-retryable error always fails the run immediately, under either policy.

Resuming a paused or failed run gives it as many more attempts as its function’s policy allows, counted from the attempt it was on. The attempt number never goes backwards.

Asking for a longer wait

Sometimes the failing service tells you when to come back. Say so, and the engine waits that long instead of using the computed backoff, up to maxDelay:

• A push endpoint that answers 429 or 503 with a Retry-After header (seconds or an HTTP date) sets the delay of the next attempt.
• A handler can return a retry-after error (RetryAfterError in each SDK) carrying a duration. The SDK sends it as retry_after_ms, in push and pull mode.

Push endpoint status codes

In push mode the engine classifies the HTTP status your endpoint answers with. 400, 401, 403, 405 and 422 fail the run at once, because resending the same request cannot fix it. Every other non-2xx status retries, 404 included, since a 404 is often a deploy still rolling out. Change the list with spec.engine.retry.nonRetryableStatuses. The full table is in the push protocol.

An HTTP response is never retried inside the request: each retry is a new durable attempt with its own backoff. Only a failure with no response at all (a refused or reset connection) is retried once, immediately.

What counts as retryable

The engine retries a failed run only when the SDK reports retryable: true on the failure. Go, TypeScript and Python use one rule, in push and pull mode, inside step.run and outside any step:

• An error that is not an IronflowError is retryable. This includes a plain Error, a Go error, a Python Exception, and a thrown non-error value.
• An IronflowError carries its own retryable flag. NonRetryableError sets it to false.
• An InvokeError (the child of step.invoke failed or timed out) is not retryable. A retry can only replay the same child failure. In Go, *InvokeError is not an IronflowError, but the rule is the same.

So throw new Error("temporary failure") uses the function’s retry policy. Compensations run only when the run is over. A non-retryable failure ends it at once. A retryable failure that exhausts its attempts ends it only under onExhausted: "fail"; under the default pause the run is not over, so nothing is compensated until it is resumed and finishes, or is cancelled. Throw NonRetryableError to stop the retries and run the saga straight away. See Sagas.

In TypeScript, a bare new IronflowError(msg) defaults to retryable: false. Pass { retryable: true } when you want it retried.

In Go, ironflow.NewNonRetryableError(msg), ironflow.WrapNonRetryable(err) and ironflow.NewError(msg, code, false) are all non-retryable. The SDK reads the flag with errors.As, so an error that you wrap with fmt.Errorf("...: %w", err) keeps it.

---

NonRetryableError

Use NonRetryableError to indicate permanent failures that shouldn’t be retried:

TypeScript

import { NonRetryableError } from "@ironflow/node";

await step.run("validate", async () => {
  if (!isValid(data)) {
    // Won't retry - permanent failure
    throw new NonRetryableError("Invalid input");
  }
  throw new Error("Temporary failure"); // will retry
});

Go

import "errors"

ironflow.Run(ctx, "validate", func() (any, error) {
    if !isValid(data) {
        // Won't retry - permanent failure
        return nil, ironflow.NewNonRetryableError("invalid input")
    }
    return nil, errors.New("temporary failure") // will retry
})

Python

from ironflow.worker import NonRetryableError


def validate():
    if not is_valid(data):
        raise NonRetryableError("invalid input", code="INVALID_INPUT")
    raise RuntimeError("temporary failure")  # will retry

await ctx.step.run("validate", validate)

When to use NonRetryableError:

• Invalid input data that won’t change on retry
• Business logic failures (e.g., insufficient funds)
• Authentication/authorization errors
• Resource not found errors

When NOT to use NonRetryableError:

• Network timeouts
• External service temporary failures
• Rate limiting (should back off and retry)

---

Error Types Summary

Error Type	Behavior	Use Case
Regular Error	Retried with backoff in every SDK — see What counts as retryable	Transient failures
NonRetryableError	Not retried	Permanent failures — invalid input, business-rule violations. In Go, ironflow.NewNonRetryableError(msg) or ironflow.WrapNonRetryable(err).
StepError	Thrown by Ironflow when a step’s underlying error propagates out of step.run; carries the wrapped error’s retryable flag	Catch around step calls to inspect step name + attempt count
StepTimeoutError *	Retryable — subject to function retry policy	A step exceeded its configured timeout
TimeoutError	Retryable — subject to function retry policy	Exported for classification. No handler or step throws it; only the consumer-group stream client throws it, when an ack gets no reply in time. A push execution exceeding its budget is cut off at the transport, not surfaced as this error.
ValidationError / SchemaValidationError	Not retried	Event payload failed schema validation
InvokeError / InvokeTimeoutError *	Not retried	step.invoke child run failed or timed out

* Thrown by the SDK but not re-exported from @ironflow/node. Import these three from @ironflow/core if you need to instanceof them.

Use the isRetryable(err) helper (isRetryable in TypeScript, exported from @ironflow/node; ironflow.IsRetryable in Go) to test how a caught error will be classified. Both return true for an unrecognised error.

---

Webhook Signature Verification

When the server has IRONFLOW_SIGNING_KEY set, it signs every push-mode request. With no key set, the server sends no signature. The SDK verifies signatures automatically when you give it the same signing key:

TypeScript

import { serve } from "@ironflow/node";

export const POST = serve({
  functions: [myFunction],
  signingKey: process.env.IRONFLOW_SIGNING_KEY, // Automatic verification
});

Note

Manual signature verification is not yet available in the JS SDK. Use the signingKey option in serve() for automatic verification.

Go

handler := ironflow.Serve(ironflow.ServeConfig{
    Functions:  []ironflow.Function{MyFunction},
    SigningKey: os.Getenv("IRONFLOW_SIGNING_KEY"), // Automatic verification
})

For manual verification:

err := ironflow.VerifySignature(
    payload,
    req.Header.Get("X-Ironflow-Signature"),
    signingKey,
    ironflow.DefaultSignatureTolerance,
)

Python

from ironflow.serve import serve

app = serve([my_function])  # verifies with IRONFLOW_SIGNING_KEY automatically

For manual verification:

from ironflow.serve import SignatureError, verify_signature

try:
    verify_signature(body, headers.get("x-ironflow-signature"), signing_key)
except SignatureError:
    ...  # reject with 401

Signature Header

The server puts the signature in the X-Ironflow-Signature header, computed with HMAC-SHA256.

Development Mode

During local development, you can skip verification:

TypeScript

export const POST = serve({
  functions: [myFunction],
  skipVerification: true, // Only for local development!
});

Go

handler := ironflow.Serve(ironflow.ServeConfig{
    Functions:        []ironflow.Function{MyFunction},
    SkipVerification: true, // Only for local development!
})

Python

from ironflow.serve import serve

# Only for local development! An empty key disables verification, even when
# IRONFLOW_SIGNING_KEY is set: the handler runs any well-formed request.
app = serve([my_function], signing_key="")

Caution

Never disable signature verification in production. This protects your endpoints from unauthorized requests.

---

Global Error Observation (Client onError)

For client-side operations (emitting events, managing runs, KV store, etc.), you can register a global onError handler to observe all errors without wrapping every call in try/catch:

import { createClient } from "@ironflow/node";

const client = createClient({
  onError: async (error, context) => {
    // Send to your logging/metrics system
    await logger.error("Ironflow client error", {
      method: context.method,        // e.g. "emit", "kv.bucket.get"
      endpoint: context.endpoint,    // e.g. "/ironflow.v1.IronflowService/Trigger"
      statusCode: context.statusCode, // HTTP status or undefined for network errors
      error: error.message,
    });
  },
});

Key behaviors:

• The handler fires before the error is re-thrown — it never suppresses errors
• Async handlers are fully awaited before the error propagates
• If the handler itself throws, its error is swallowed (logged to stderr)
• Propagates to sub-clients created via client.kv() and client.config()

In Python, pass on_error to IronflowClient, IronflowRPC or AsyncIronflowRPC:

import logging

from ironflow import IronflowClient

log = logging.getLogger("app")


def on_error(error, ctx):
    log.error("ironflow client error", extra={
        "method": ctx.method,            # HTTP verb (REST) or RPC method name
        "endpoint": ctx.endpoint,        # "/api/v1/events" or "/ironflow.v1.PubSubService/Publish"
        "status_code": ctx.status_code,  # None for network errors and on the RPC clients
        "error": str(error),
    })


client = IronflowClient(on_error=on_error)

The hook fires once per failed call, after retries are exhausted, and never suppresses the error. A hook that raises is logged and swallowed. AsyncIronflowRPC also accepts an async def hook and awaits it.

The hook does not apply to KV and config watch streams. Do not call the same client from inside the hook: a failing call would call the hook again. str(error) can include request details, so do not log it blindly if you use secret query parameters.

This is useful for centralized logging, metrics collection, and alerting on client errors. See the @ironflow/node reference for the full API.

The Go client has no client-wide hook. In Go, set WorkerConfig.OnError to observe jobs that fail inside a worker; it runs after compensations and before the failure is reported, and it does not change retry behavior.

Tip

onError is for observing client errors. For controlling retry behavior inside functions, use NonRetryableError instead.

---

Handling Failed Runs

When a run fails after exhausting retries, you can:

1. Hot Patch: Edit step outputs and resume from a specific step
2. Investigate: Use the TUI debugger or dashboard to inspect the failure
3. Fix and Retry: Fix the underlying issue and trigger a new event

See Debugging for more details on investigating and recovering from failures.

Reading the earlier attempts

Each retry records the attempt it replaces, so the error of attempt 1 is still there when attempt 5 fails. This works for every function, with or without recording.

• Dashboard. Expand a step on the run page. A step that ran more than once lists its attempts with the outcome, start time, duration, worker and error. The duration is the one the worker measured when it reports one (push, and the Go, Node and Python pull workers do); otherwise it is the time between the reports the engine received. An earlier attempt with no end is marked abandoned: its worker stopped reporting and a later execution ran the step again. The run page has a Retries tab with the error of each failed execution the engine retried.
• API. GetRunSteps returns attempts on each step, and GetRun returns the run-level retries in attempts. See Attempt history.

The engine keeps the first attempt and the last 10 of each step, which covers the cause and the end of a long retry loop; the attempts in between are dropped. The history is deleted with its run. Redacting the step or the run replaces each attempt’s error with the redaction placeholder; redacting a step also scrubs the run’s retries, which carry the step’s error.

---

What’s Next?

• Debugging — Hot patching, TUI debugger, VS Code DAP
• API Reference — REST API endpoints