Circuit Breakers

Circuit breakers protect push mode endpoints from cascading failures. When a function’s HTTP endpoint returns repeated errors, the circuit opens to stop sending requests. After a timeout, a single probe request tests recovery. If it succeeds, the circuit closes and normal traffic resumes; if it fails, the open window re-arms.

How It Works

Each push mode function gets its own circuit breaker, keyed by function ID (the stored endpoint is informational — it lets the HTTP/CLI layer rebuild the legacy fnID|endpoint key). Two functions sharing the same endpoint URL therefore get independent breakers.

  ┌──────────┐  5 consecutive  ┌──────────┐  60s elapsed   ┌───────────┐
  │  CLOSED  │───failures─────▶│   OPEN   │───────────────▶│ HALF-OPEN │
  │ (normal) │                 │ (reject) │                │  (probe)  │
  └──────────┘                 └──────────┘                └───────────┘
       ▲                            ▲                        │         │
       │  success                   │  failure               │         │
       └────────────────────────────┴────────────────────────┘         │
                                                 success ──────────────┘

States:

State	Behavior
Closed	Normal operation. Requests pass through to the endpoint.
Open	Failing fast. All requests are rejected without calling the endpoint. Runs are deferred for retry.
Half-Open	A single probe request is allowed through (claimed cluster-wide). If it succeeds, the circuit closes; if it fails, the open window re-arms for another timeout.

Default thresholds:

Setting	Default	Description
Failure threshold	5	Consecutive failures before opening the circuit
Success threshold	1	Consecutive successes in half-open before closing
Timeout	60 seconds	How long the circuit stays open before probing

State Persistence

Circuit breaker state is persisted in the database (circuit_breaker_states table) and evaluated at dispatch reservation time, in the same transaction as the queue + capacity decision. This means:

• Restart survival: A restarted node inherits open circuits — it reads breaker state from the database at dispatch time instead of allowing traffic to endpoints that were failing.
• Cross-node consistency: In a multi-node cluster, breaker state is a single database row. The moment one node opens a circuit, every node’s dispatch reservation sees it — there is no propagation delay.
• Rolling deploy safety: State carries across rolling deploys without protection gaps.

An open circuit blocks new push reservations; in-flight executions continue. When the open window elapses, the dispatch reservation claims exactly one probe request per cluster (half-open) by transitioning the breaker to a stored half_open state — a single cluster-wide winner, not a burst. A probe success closes the circuit and the backlog drains immediately; a failed probe re-opens and re-arms the window. If the probe runner crashes without reporting, the next window re-claims a fresh probe, so the breaker never wedges open. A fully-down endpoint has no successful probe to close it and stays protected.

Viewing Circuit Breaker State

Dashboard

The Functions page shows a circuit breaker state badge next to each function. Open circuits show a red badge, half-open shows yellow. Closed circuits show no badge (normal state).

CLI

# List all circuit breakers
ironflow circuit-breaker list

# JSON output
ironflow circuit-breaker list --json

Example output:

FUNCTION_ID    ENDPOINT                           STATE   FAILS  LAST_FAILURE
fn-payments    http://payments:3000/api/ironflow   open    5      2026-04-06T12:00:00Z
fn-orders      http://orders:4000/api/ironflow     closed  0      -

API

# List all breaker states
curl http://localhost:9123/api/v1/circuit-breakers

# Response
[
  {
    "key": "fn-payments|http://payments:3000/api/ironflow",
    "function_id": "fn-payments",
    "endpoint": "http://payments:3000/api/ironflow",
    "state": "open",
    "consecutive_fails": 5,
    "last_failure": "2026-04-06T12:00:00Z"
  }
]

Prometheus Metrics

The ironflow_circuit_breaker_state gauge carries two breakers. Its function_id label holds the breaker key, which is a real function ID for the per-function push-endpoint breaker and the literal nats-publish for the shared NATS publish breaker that guards event publishing process-wide. The state label is one of closed, open, or half_open. The value is 1 on the series matching the breaker’s current state and 0 on the other two, so filter on the label and compare to 1.

# All open push-endpoint circuits
ironflow_circuit_breaker_state{function_id!="nats-publish", state="open"} == 1

# The shared NATS publish circuit
ironflow_circuit_breaker_state{function_id="nats-publish", state="open"} == 1

Two alerts ship with the Helm chart (deploy/helm/ironflow/templates/ironflow-alerts.yaml), split on that same label so each names the breaker it actually watches: PushEndpointCircuitOpen for a function’s endpoint, NATSPublishCircuitOpen for the shared publish breaker. Both fire after 2 minutes of continuous open state. Bare-binary and docker-compose deploys must wire these alerts themselves.

Resetting a Circuit Breaker

If you’ve fixed the downstream issue and don’t want to wait for the 60-second timeout, you can manually reset a circuit breaker:

CLI

# Reset by endpoint URL
ironflow circuit-breaker reset https://payments:3000/api/ironflow

# Reset by function ID
ironflow circuit-breaker reset fn-payments

The arg is detected as an endpoint URL if it contains ://, otherwise treated as a function ID.

API

# The key is the base64url-encoded composite key (function_id|endpoint_url)
KEY=$(echo -n "fn-payments|http://payments:3000/api/ironflow" | base64 | tr '+/' '-_' | tr -d '=')
# Endpoints with shell-special chars: keep the value quoted as shown above.
curl -X POST http://localhost:9123/api/v1/circuit-breakers/$KEY/reset

How Circuit Breakers Interact with Other Features

Retry scheduling: When a circuit is open, retries are blocked at dispatch reservation rather than attempted — they stay queued until the breaker’s next_probe_at elapses (60s after opening, by default). This prevents wasting retry attempts against a known-failing endpoint.

Cron triggers: Cron-triggered runs are skipped entirely when the circuit for their function’s endpoint is open. The cron scheduler logs a debug message and moves on to the next scheduled time.

Multi-node clusters: Circuit breaker state is a single database row, consulted by every node’s dispatch reservation. The failure counter is shared, so the circuit opens after failure_threshold failures across the whole cluster (not per-node), and once open every node stops dispatching to the endpoint with no propagation delay.