Circuit Breakers

Circuit breakers protect push mode endpoints from cascading failures. When a function’s HTTP endpoint returns repeated errors, the circuit opens to stop sending requests. After a timeout, a single probe request tests recovery. If it succeeds, the circuit closes and normal traffic resumes; if it fails, the open window re-arms.

How It Works

Each push mode function gets its own circuit breaker, keyed by function ID and endpoint URL. This means two functions sharing the same endpoint URL get independent breakers.

  ┌──────────┐  5 consecutive  ┌──────────┐  60s elapsed   ┌───────────┐
  │  CLOSED  │───failures─────▶│   OPEN   │───────────────▶│ HALF-OPEN │
  │ (normal) │                 │ (reject) │                │  (probe)  │
  └──────────┘                 └──────────┘                └───────────┘
       ▲                            ▲                        │         │
       │  success                   │  failure               │         │
       └────────────────────────────┴────────────────────────┘         │
                                                 success ──────────────┘

States:

State	Behavior
Closed	Normal operation. Requests pass through to the endpoint.
Open	Failing fast. All requests are rejected without calling the endpoint. Runs are deferred for retry.
Half-Open	A single probe request is allowed through (claimed cluster-wide). If it succeeds, the circuit closes; if it fails, the open window re-arms for another timeout.

Default thresholds:

Setting	Default	Description
Failure threshold	5	Consecutive failures before opening the circuit
Success threshold	1	Consecutive successes in half-open before closing
Timeout	60 seconds	How long the circuit stays open before probing

State Persistence

Circuit breaker state is persisted in the database (circuit_breaker_states table) and evaluated at dispatch reservation time, in the same transaction as the queue + capacity decision. This means:

• Restart survival: A restarted node inherits open circuits — it reads breaker state from the database at dispatch time instead of allowing traffic to endpoints that were failing.
• Cross-node consistency: In a multi-node cluster, breaker state is a single database row. The moment one node opens a circuit, every node’s dispatch reservation sees it — there is no propagation delay.
• Rolling deploy safety: State carries across rolling deploys without protection gaps.

An open circuit blocks new push reservations; in-flight executions continue. When the open window elapses, the dispatch reservation claims exactly one probe request per cluster (half-open) by transitioning the breaker to a stored half_open state — a single cluster-wide winner, not a burst. A probe success closes the circuit and the backlog drains immediately; a failed probe re-opens and re-arms the window. If the probe runner crashes without reporting, the next window re-claims a fresh probe, so the breaker never wedges open. A fully-down endpoint has no successful probe to close it and stays protected.

Viewing Circuit Breaker State

Dashboard

The Functions page shows a circuit breaker state badge next to each function. Open circuits show a red badge, half-open shows yellow. Closed circuits show no badge (normal state).

CLI

# List all circuit breakers
ironflow circuit-breaker list

# JSON output
ironflow circuit-breaker list --json

Example output:

FUNCTION_ID    ENDPOINT                           STATE   FAILS  LAST_FAILURE
fn-payments    http://payments:3000/api/ironflow   open    5      2026-04-06T12:00:00Z
fn-orders      http://orders:4000/api/ironflow     closed  0      -

API

# List all breaker states
curl http://localhost:9123/api/v1/circuit-breakers

# Response
[
  {
    "key": "fn-payments|http://payments:3000/api/ironflow",
    "function_id": "fn-payments",
    "endpoint": "http://payments:3000/api/ironflow",
    "state": "open",
    "consecutive_fails": 5,
    "last_failure": "2026-04-06T12:00:00Z"
  }
]

Prometheus Metrics

The ironflow_circuit_breaker_state gauge tracks circuit breaker state per function. The state Prometheus label is one of closed, open, or half-open; the gauge value also encodes the state numerically (0=closed, 1=open, 2=half-open), so either form works for filtering.

# Find all open circuits
ironflow_circuit_breaker_state{state="open"} == 1

The NATSPublishCircuitOpen alert ships with the Helm chart (deploy/helm/ironflow/templates/ironflow-alerts.yaml) and fires after 2 minutes of continuous open state. Bare-binary and docker-compose deploys must wire this alert themselves.

Resetting a Circuit Breaker

If you’ve fixed the downstream issue and don’t want to wait for the 60-second timeout, you can manually reset a circuit breaker:

CLI

# Reset by endpoint URL
ironflow circuit-breaker reset https://payments:3000/api/ironflow

# Reset by function ID
ironflow circuit-breaker reset fn-payments

The arg is detected as an endpoint URL if it contains ://, otherwise treated as a function ID.

API

# The key is the base64url-encoded composite key (function_id|endpoint_url)
KEY=$(echo -n "fn-payments|http://payments:3000/api/ironflow" | base64 | tr '+/' '-_' | tr -d '=')
# Endpoints with shell-special chars: keep the value quoted as shown above.
curl -X POST http://localhost:9123/api/v1/circuit-breakers/$KEY/reset

How Circuit Breakers Interact with Other Features

Retry scheduling: When a circuit is open, the scheduler defers retries by 60 seconds instead of attempting them immediately. This prevents wasting retry attempts against a known-failing endpoint.

Cron triggers: Cron-triggered runs are skipped entirely when the circuit for their function’s endpoint is open. The cron scheduler logs a debug message and moves on to the next scheduled time.

Multi-node clusters: Circuit breaker state is a single database row, consulted by every node’s dispatch reservation. The failure counter is shared, so the circuit opens after failure_threshold failures across the whole cluster (not per-node), and once open every node stops dispatching to the endpoint with no propagation delay.