Ironflow is pre-1.0: APIs can change between releases. Pin a version. What is ready →

Circuit Breakers

Circuit breakers protect push mode endpoints from cascading failures. When a function’s HTTP endpoint returns repeated errors, the circuit opens to stop sending requests. After a timeout, a single probe request tests recovery. If it succeeds, the circuit closes and normal traffic resumes; if it fails, the circuit re-opens for a longer window than the last one.

How It Works

Each push mode function gets its own circuit breaker, keyed by environment and function ID (the stored endpoint is informational — it lets the HTTP/CLI layer rebuild the legacy fnID|endpoint key). Two functions sharing the same endpoint URL therefore get independent breakers. Two environments that use the same function ID also get independent breakers. A reset acts on the caller’s environment; a platform key must send X-Ironflow-Environment-ID.

  ┌──────────┐  5 consecutive  ┌──────────┐  window over   ┌───────────┐
  │  CLOSED  │───failures─────▶│   OPEN   │───────────────▶│ HALF-OPEN │
  │ (normal) │                 │ (reject) │                │  (probe)  │
  └──────────┘                 └──────────┘                └───────────┘
       ▲                            ▲                        │         │
       │  success                   │  failure               │         │
       └────────────────────────────┴────────────────────────┘         │
                                                 success ──────────────┘

States:

State	Behavior
Closed	Normal operation. Requests pass through to the endpoint.
Open	Failing fast. Ironflow does not call the endpoint. A new run stays queued in `waiting_for_capacity`, and a run waiting to retry shows `waiting`; nothing is dropped or failed by the breaker, and a held run does not use up an attempt.
Half-Open	A single probe request is allowed through (claimed cluster-wide). If it succeeds, the circuit closes; if it fails, the circuit re-opens for double the previous window, up to the maximum.

Policy

Setting	Default	Description
`failureThreshold`	5	Consecutive failures before opening the circuit
`openDuration`	60s	How long the circuit stays open before the first probe
`maxOpenDuration`	10m	Cap on the open window as it grows (see below)

One probe success closes the circuit.

Each setting is resolved in three layers, and a layer only overrides the fields it sets:

The built-in defaults above.

The server config, spec.engine.circuitBreaker (applies to every function):

spec:
  engine:
    circuitBreaker:
      failureThreshold: 5
      openDuration: 60s
      maxOpenDuration: 10m

The function’s own settings, sent at registration as circuit_breaker (failure_threshold, open_duration_ms, max_open_duration_ms). A zero field keeps the value from the layer below.

The SDKs take the same override on the function definition. A zero or omitted field keeps the inherited value:

// Node
createFunction(
  {
    id: "fn-payments",
    triggers: [{ event: "payment.requested" }],
    circuitBreaker: { failureThreshold: 3, openDurationMs: 30_000, maxOpenDurationMs: 300_000 },
  },
  async ({ event, step }) => { /* ... */ },
);

// Go
ironflow.CreateFunction(ironflow.FunctionConfig{
    ID:       "fn-payments",
    Triggers: []ironflow.Trigger{{Event: "payment.requested"}},
    CircuitBreaker: &ironflow.CircuitBreakerConfig{
        FailureThreshold: 3,
        OpenDuration:     30 * time.Second,
        MaxOpenDuration:  5 * time.Minute,
    },
}, handler)

# Python
@function(
    id="fn-payments",
    triggers=[{"event": "payment.requested"}],
    circuit_breaker={"failure_threshold": 3, "open_duration": "30s", "max_open_duration": "5m"},
)
async def fn_payments(ctx): ...

Registration rejects a function that sets both open_duration_ms and max_open_duration_ms with the maximum below the open duration. The server config does not reject that pair: the engine raises the cap to the window instead, so a function that raises open_duration_ms above the inherited maximum simply does not escalate. The server config is validated at startup (ironflow validate reports it): failureThreshold must be at least 1. In the server config, openDuration and maxOpenDuration may be at most 24h; a larger value is rejected. A function’s own millisecond fields are 32-bit, so they top out at about 24.8 days and have no 24h cap.

Minimum window. The open window is never shorter than 2 × pushTimeout + 0.3s + 1s (21.3s with the default 10s push timeout), the longest one probe can run: two request budgets and the single inline retry between them. A shorter openDuration is raised to that floor rather than rejected, and maxOpenDuration is raised with it if it would fall below. The floor guarantees a probe has finished, one way or the other, before the gate can claim a second one; without it a slow first probe could re-open the circuit a second time or release the newer probe.

The policy is read when a failure is recorded, so a change applies to the next failure. A breaker that is already open keeps the window it was armed with.

Escalating open window

A probe that fails re-opens the circuit for twice the window it just served, capped at maxOpenDuration. With the defaults the waits between probes are 60s, 2m, 4m, 8m, then 10m from there on. A dead endpoint is probed less and less often instead of once a minute forever. Closing the circuit, by a successful probe or by a manual reset, starts the next opening at openDuration again.

The window is stored on the breaker’s database row, so the value a failure records is the one every node waits out, and the dispatch reservation that claims a probe re-arms its crash-recovery deadline from the same value. The policy that computes it comes from the node that records the failure, so keep spec.engine.circuitBreaker (and pushTimeout, which sets the minimum window) the same in every node’s YAML; nodes with different files record different windows.

What Counts as a Failure

The breaker watches the endpoint, not your handler’s logic. A failure is any of:

a connection error or a timeout (the push timeout is 10 seconds by default)
an HTTP 4xx or 5xx response, including 404
a 2xx response whose body is not valid JSON

A handler that throws does not count. The Node, Go and Python SDKs catch the error and answer 200 with status: "failed", so the endpoint is healthy from the breaker’s point of view and the run follows its normal retry policy. A crash outside the handler that makes the SDK answer 500 does count.

Failures are counted per durable attempt, after any immediate transport retry within that attempt. A run that keeps failing goes back to the queue for a durable retry and counts once per attempt, so one run can contribute up to its retry budget toward the threshold.

State Persistence

Circuit breaker state is persisted in the database (circuit_breaker_states table) and evaluated at dispatch reservation time, in the same transaction as the queue + capacity decision. This means:

Restart survival: A restarted node inherits open circuits — it reads breaker state from the database at dispatch time instead of allowing traffic to endpoints that were failing.
Cross-node consistency: In a multi-node cluster, breaker state is a single database row. The moment one node opens a circuit, every node’s dispatch reservation sees it — there is no propagation delay.
Rolling deploy safety: State carries across rolling deploys without protection gaps.

An open circuit blocks new push reservations; in-flight executions continue. When the open window elapses, the dispatch reservation claims exactly one probe request per cluster (half-open) by transitioning the breaker to a stored half_open state — a single cluster-wide winner, not a burst. A probe success closes the circuit and the backlog drains immediately; a failed probe re-opens the circuit with a doubled window. If the probe runner crashes without reporting, the next window re-claims a fresh probe, so the breaker never wedges open. A probe whose run is cancelled or held before the request is sent gives no health signal, so it is handed back at once and the next ready run claims a fresh probe. A fully-down endpoint has no successful probe to close it and stays protected.

Viewing Circuit Breaker State

Dashboard

The Functions page shows a circuit breaker state badge next to each function. Open circuits show a red badge, half-open shows yellow. Closed circuits show no badge (normal state). Hover the badge for the endpoint, the consecutive failure count, the last failure time, and the last failure reason. The Reset button next to the badge closes the breaker.

The Overview page has a Circuit Breakers card with the number of open breakers. It counts only open; a half-open breaker is not counted.

Events

Each state change publishes system.circuit_breaker.{state}.{functionId} with the from and to states and the consecutive failure count. Subscribe to it to alert on an opening breaker. See system events.

CLI

# List all circuit breakers
ironflow circuit-breaker list

# JSON output
ironflow circuit-breaker list --json

Example output:

ENVIRONMENT  FUNCTION_ID  ENDPOINT                            STATE   FAILS  LAST_FAILURE
env_default  fn-payments  http://payments:3000/api/ironflow   open    5      2026-04-06T12:00:00Z
env_default  fn-orders    http://orders:4000/api/ironflow     closed  0      -

API

# List all breaker states
curl http://localhost:9123/api/v1/circuit-breakers

# Response
[
  {
    "key": "fn-payments|http://payments:3000/api/ironflow",
    "environment_id": "env_default",
    "function_id": "fn-payments",
    "endpoint": "http://payments:3000/api/ironflow",
    "state": "open",
    "consecutive_fails": 5,
    "last_failure": "2026-04-06T12:00:00Z",
    "last_failure_reason": "http_server_error"
  }
]

last_failure_reason classifies the latest recorded failure. It is one of http_client_error, http_server_error, timeout, connection_error, invalid_response, or transport_error. It never contains an error message, response body, URL, or credential. The endpoint field still contains the push URL.

Every recorded failure updates the reason and timestamp together, including a failure from a request that started before the circuit opened. A successful probe, manual reset, or released probe retains this history. The reset response returns the retained history too. Rows created before reason tracking omit the reason until their next failure; the dashboard shows an em dash for those rows. The MCP ironflow_circuit_breaker_list tool returns the same history fields.

Prometheus Metrics

The ironflow_circuit_breaker_state gauge carries two breakers. Its function_id label holds the breaker key, which is a real function ID for the per-function push-endpoint breaker and the literal nats-publish for the shared NATS publish breaker that guards event publishing process-wide. Its environment_id label names the environment of a function breaker, because a function ID is only unique inside one environment. The nats-publish series has an empty environment_id. Deleting a function or an environment removes its series, so an open breaker on a deleted function does not keep an alert firing. The state label is one of closed, open, or half_open. The value is 1 on the series matching the breaker’s current state and 0 on the other two, so filter on the label and compare to 1.

# All open push-endpoint circuits
ironflow_circuit_breaker_state{function_id!="nats-publish", state="open"} == 1

# The shared NATS publish circuit
ironflow_circuit_breaker_state{function_id="nats-publish", state="open"} == 1

Ironflow ships no alert rules, so wire your own. Write one alert per expression above, so each alert names the breaker it watches, and fire it after 2 minutes of continuous open state.

Resetting a Circuit Breaker

If you’ve fixed the downstream issue and don’t want to wait for the open window to end, you can manually reset a circuit breaker. A reset closes the circuit, clears the escalated window, sets the ironflow_circuit_breaker_state gauge to closed and publishes the system.circuit_breaker.closed.{functionId} event, the same as a probe success would. In a multi-node cluster each node keeps its own gauge: the node that handled the reset updates it at once, and every other node converges on its next capacity scanner sweep (every 30 seconds), when it re-reads the breaker rows from the database.

CLI

# Reset by endpoint URL
ironflow circuit-breaker reset https://payments:3000/api/ironflow

# Reset by function ID
ironflow circuit-breaker reset fn-payments

The arg is detected as an endpoint URL if it contains ://, otherwise treated as a function ID. --env (-e) selects the environment (default: IRONFLOW_ENV). A platform key must pass it.

API

# The key is the base64url-encoded composite key (function_id|endpoint_url)
KEY=$(echo -n "fn-payments|http://payments:3000/api/ironflow" | base64 | tr '+/' '-_' | tr -d '=\n')
# Endpoints with shell-special chars: keep the value quoted as shown above.
curl -X POST http://localhost:9123/api/v1/circuit-breakers/$KEY/reset

How Circuit Breakers Interact with Other Features

Retry scheduling: When a circuit is open, retries are blocked at dispatch reservation rather than attempted — they stay queued until the breaker’s next_probe_at elapses (openDuration after opening, 60s by default, longer after failed probes). This prevents wasting retry attempts against a known-failing endpoint.

Cron triggers: Cron-triggered runs are skipped entirely when the circuit for their function’s endpoint is open. The cron scheduler logs a debug message and moves on to the next scheduled time.

Multi-node clusters: Circuit breaker state is a single database row, consulted by every node’s dispatch reservation. The failure counter is shared, so the circuit opens after failure_threshold failures across the whole cluster (not per-node), and once open every node stops dispatching to the endpoint with no propagation delay.