Observability

Ironflow provides built-in observability through OpenTelemetry distributed tracing and Prometheus metrics. Both are optional — zero overhead when disabled.

Key Concepts

Feature	Description
Analytics page	Built-in dashboard charts for runs, events and run duration. No setup
Prometheus Metrics	Counter, histogram, and gauge metrics for runs, steps, events, and HTTP requests
OpenTelemetry Tracing	Distributed trace spans for push segments, pull assignment, and HTTP requests
W3C Trace Context	Automatic propagation of traceparent/tracestate headers
Zero Overhead	No performance impact when observability is disabled

Configuration

Configure observability via environment variables:

Variable	Description	Default
IRONFLOW_METRICS_ENABLED	Enable Prometheus metrics at /metrics	false
IRONFLOW_OTEL_ENDPOINT	OTLP gRPC endpoint for tracing (empty = disabled)	(empty)
IRONFLOW_OTEL_SAMPLE_RATE	Trace sampling rate (0.0 to 1.0)	1.0
IRONFLOW_OTEL_SERVICE_NAME	Service name in trace data	ironflow
IRONFLOW_OTEL_INSECURE	Use plaintext gRPC for OTLP export (set false for TLS)	true

Every one of these has a spec.observability equivalent in ironflow.yaml. A field set in the file wins over the environment variable; a field the file leaves out keeps its environment value.

Analytics Page

The dashboard has an Analytics page at /analytics. It reads the Ironflow database, so it needs no metrics, Prometheus or Grafana. It is always on.

The page shows data for the current environment only:

• Totals: run count, error rate, and P50 and P95 run duration.
• Run throughput: runs per time bucket, stacked by outcome. Runs that are not finished show as a grey hatched segment.
• Event ingest: events per time bucket, stacked by kind: app events, entity stream events and system events.
• Functions: runs, failures, error rate, P50, P95 and last failure for each function. Failures sort first. Each row links to the Runs page with the same filter. The table shows the top 25 functions until you select Show all.

Select a range: 1h, 24h (default), 7d or 30d. The page refreshes every 30 seconds. The server caches each environment and range for 60 seconds, so a chart can be up to 60 seconds old.

Things to know when you read the numbers:

• Duration is wall-clock time from run start to run end, for completed and failed runs. It includes sleep and wait-for-event time, so a function that sleeps for an hour has a P95 of an hour or more.
• Error rate is failed ÷ (completed + failed). Cancelled runs and runs that are not finished are not in the rate.
• Retention: if IRONFLOW_RUN_RETENTION_DAYS, IRONFLOW_EVENT_RETENTION_DAYS or IRONFLOW_ENTITY_EVENT_RETENTION_DAYS is shorter than the range, the page shows a notice. Old rows can be pruned, so that series may be incomplete.
• Permission: the page needs the functions:list permission in the environment.

The API is GET /api/v1/analytics?range=1h|24h|7d|30d. For HTTP latency, NATS metrics, alerts or history longer than your retention, use Prometheus metrics (below).

Prometheus Metrics

When IRONFLOW_METRICS_ENABLED=true, the /metrics endpoint serves Prometheus exposition format on the main API port by default. Pass --metrics-port, or set YAML spec.observability.metrics.port, to a distinct port (1–65535) to use a dedicated listener on the configured server host. Zero/unset preserves the default; disabled metrics starts no additional listener.

The registry includes Go runtime metrics (go_*), process metrics (process_*), and local database pool gauges on SQLite and PostgreSQL. The pool gauges are ironflow_db_pool_open_connections, ironflow_db_pool_in_use_connections, ironflow_db_pool_idle_connections, and ironflow_db_pool_max_connections. A maximum of zero means unlimited. Pool scrapes read memory without acquiring a database connection.

Trace resources include service.version from the binary build and service.instance.id from the resolved engine node ID. The Cloud control plane uses a generated ID for its process lifetime because it has no engine node.

Core Metrics

The table lists common operator metrics, including every metric queried by a bundled alert. Scrape /metrics for the full registry.

Metric	Type	Labels	Description
ironflow_outbox_pending_count	Gauge	None	Pending outbox entries across environments. Use max() across nodes.
ironflow_outbox_oldest_pending_age_seconds	Gauge	None	Oldest pending entry age at sampling time. Use max() across nodes.
ironflow_nats_consumer_lag	Gauge	stream	Sum of undelivered pending messages across consumer groups in each engine stream. Use max by (stream) across nodes.
ironflow_jetstream_stream_leader_missing_seconds	Gauge	stream	Seconds the engine stream has answered with no leader, not answered, or not been listed; 0 otherwise, also for replaced and config pending. Use max by (stream) across nodes. See Recover a stuck JetStream stream.
ironflow_jetstream_stream_fault	Gauge	stream, fault	1 while /ready reports the fault for the stream: no leader, no answer, missing, replaced or config pending. There is no series while the stream has no fault. The JetStreamStreamFault alert reads it with max by (stream, fault).
ironflow_runs_total	Counter	function_id, status, environment_id, failure_cause	Total workflow runs. failure_cause is platform or user for failed runs, empty for completed.
ironflow_runs_paused_exhausted_total	Counter	function_id, environment_id	Runs paused because they used up their retry budget under onExhausted: pause (the default). A paused run is not failed, so status="failed" and HighRunFailureRate never count it. The overview’s stats.paused_exhausted_runs is how many sit paused now; ironflow run resume <run-id> grants a fresh budget.
ironflow_run_duration_seconds	Histogram	function_id, environment_id	Run execution duration
ironflow_steps_total	Counter	function_id, status, environment_id	Finished steps, one count per outcome. status is completed, failed, or timed_out (a waitForEvent or step.invoke() that hit its timeout). Each failed attempt counts once. A memoized step replayed on resume does not count again, and neither does a step a worker reports again, with one exception: in REST pull mode, a failed step in a checkpoint batch that the worker resends (after a partial store error or a lost 2xx response) counts again. Not labeled by step ID: step IDs are yours to choose and are commonly interpolated with request data (step.run(`search-${query}`)), which would make the label unbounded. Use the runs API or your traces for a per-step breakdown.
ironflow_step_retries_total	Counter	environment_id, function_id	Successfully committed newer durable-step attempts across push, REST and streaming workers. Duplicate reports, lost CAS writes, initial or adopted attempts and run-level outage retries do not increment it.
ironflow_push_dispatch_duration_seconds	Histogram	environment_id, function_id, outcome	Each push HTTP attempt through response-body handling, including inline retries. Outcomes are 2xx, 4xx, 5xx, timeout and conn_error.
ironflow_step_duration_seconds	Histogram	function_id, environment_id	How long a step.run or a compensation step ran, one sample per such step counted above. Sleep, waitForEvent and step.invoke() steps are counted in ironflow_steps_total but leave no sample here: their time is spent waiting, can last days, and would sit past the top bucket (300 s). Not labeled by step ID, as above.
ironflow_events_emitted_total	Counter	event_name, environment_id	Events published to the pub/sub stream: your emit() events under their own name (counted when the outbox publishes them), and engine lifecycle frames (run.created, step.completed, …) under run.* / step.*. Filter on event_name to separate the two. Event names are trigger-matched identifiers — keep them static; a name interpolated with request data (emit(`order.${id}`)) adds one series per value.
ironflow_active_runs	Gauge	function_id, environment_id	Currently active runs
ironflow_http_requests_total	Counter	method, path, status_code	HTTP requests. path is the matched route pattern (/api/v1/events/{id}), or unmatched when no route matched — never the raw request path, so cardinality is bounded by the route table. A request whose handler panics is counted with status_code="500". For gRPC and gRPC-Web the label is not the wire status line: those protocols answer 200 and put the error in trailers, so status_code is derived from Grpc-Status and matches what the same error returns over Connect — a shed write is 429 whichever protocol asked. A request rejected before the router runs (a throttled write) carries no readable status on every protocol, so the rejecting middleware stamps its code instead. On a gRPC or gRPC-Web stream, Canceled and DeadlineExceeded stay 200 — that is a client disconnecting or timing out, not a failure — while every other code counts, so a stream that dies of Internal is not recorded as a success. A Connect-protocol stream is the exception: it carries no Grpc-Status to read, so its errors still count as 200. A request that middleware refuses before the router (authentication, the pre-route RBAC check, the environment-selector check) is counted too: a 401 for a bad credential or a 403 for a refusal is labeled with the route it was aimed at, as the route table resolves it, or unmatched when it names no route. A rejection by a handler counts under its own route: a failed dashboard or platform login under /api/v1/auth/login or /api/v1/platform/auth/login, a bad webhook token or signature under /api/v1/webhooks/{provider}. To watch refused credentials, use ironflow_auth_failures_total below.
ironflow_http_request_duration_seconds	Histogram	method, path	HTTP request duration. Same path label as above.
ironflow_auth_failures_total	Counter	reason	Requests refused for their credential, one count per request. reason is one of the classes in Authentication failures: no_credential, unknown_key, expired_key, expired_token, bad_token, revoked_session, inactive_user, bad_login, bad_webhook_token, bad_signature, wrong_transport or env_forbidden. A dashboard page or GET /api/v1/auth/validate request with no session at all is the normal logged-out state and is not counted, and neither is a database error during a login lookup. No other label: never the key, the path or the caller’s address. Every reason is exported from boot at 0. A rise in unknown_key or bad_login is credential guessing; a rise in expired_key, expired_token or revoked_session is usually a client running with stale credentials. The log line for each refusal is rate limited; this counter is not. A refusal by a known principal also writes an audit row (Refused credentials).
ironflow_workers_connected	Gauge	—	Currently connected pull-mode workers
ironflow_worker_disconnects_total	Counter	reason	Worker disconnections by reason
ironflow_outbox_dead_letter_count	Gauge	environment_id	Latest sampled rows in the outbox dead-letter table per environment. Use max by (environment_id) across nodes. See Transactional outbox.
ironflow_outbox_dlq_collector_errors_total	Counter	—	Sampling failures of the DLQ count query. A failed refresh omits its gauges; an absent series does not mean an empty DLQ.
ironflow_dlq_writes_total	Counter	source	DLQ moves since process start. source="outbox" fires every time the outbox worker exhausts the retry budget (default 10 failed publishes of the row’s own) and moves a row to outbox_dead_letter. A NATS outage normally moves no rows, so this stays flat through one: watch ironflow_outbox_pending_count and ironflow_outbox_oldest_pending_age_seconds.
ironflow_circuit_breaker_state	Gauge	environment_id, function_id, state	Breaker state by environment and function. One series exists per state (closed, open, half_open): the current state has value 1, the others 0. An open breaker blocks dispatch to that function.
ironflow_codec_encode_errors_total	Counter	reason	SubjectCodec encode failures (invalid_namespace, empty_topic, length_exceeded, wildcard_in_user_input, panic).
ironflow_codec_decode_errors_total	Counter	namespace, reason	SubjectCodec decode failures (malformed, unknown_namespace, empty_body, panic). Silent message drop on the projection consumer path — projections lag.
ironflow_capacity_leases_expired_total	Counter	environment_id, function_id	Concurrency leases expired and reclaimed by the scanner.
ironflow_capacity_orphan_runs_total	Counter	environment_id, function_id, outcome	Runs left waiting for capacity with no queued segment and no lease (the node died between creating the run and queuing it), settled by the orphan sweep: readmitted, failed (its function or event is gone, or it cannot execute) or cancelled (an invoke child no parent step links, invoke-link-failed, or whose parent has ended, parent-ended). A non-zero value means a run was never queued: a node died mid-admission, or a dispatch failed and its failure write failed too. See Crash recovery.
ironflow_throttle_delayed_total	Counter	environment_id, function_id	New runs whose start was pushed back (once per run, at its first deferral) because their function’s throttle was full. The run is delayed, never dropped. A steadily rising value means the throttle is the bottleneck for that function.
ironflow_capacity_oldest_queue_age_seconds	Gauge	environment_id, function_id	Age of the oldest waiting dispatch_queue row. Global snapshot — aggregate with max by(environment_id, function_id).
ironflow_audit_write_failures_total	Counter	source, event_type	Audit rows that failed to write or were dropped under load. A label set has no series until its first failure. See Audit rows can drop under load for the alert expression.
ironflow_retention_rows_deleted_total	Counter	retention_job	Rows the nightly retention pruner deleted, by job (audit, runs, events, entity_events, dead_letters). Successful passes only, so it measures work done, not attempts. A job whose pass deletes nothing creates no series.
ironflow_retention_consecutive_failures	Gauge	retention_job	Consecutive failed nightly retention passes for a job; reset to 0 by the next success. This is the alertable one: a wedged prune otherwise produces a single ERROR line a day and nothing else, and the first sign is disk usage. Alert on > 1 — one failure is a transient, two in a row means retention has stopped. The series exists from the first pass, so a job that has never succeeded still reports.
ironflow_policy_set_compile_failed	Gauge	org	1 when the org’s stored policy set does not compile on this node, else 0. Layer 2 then denies every request for the org (enforce) or records would_deny (monitor). The bundled IronflowPolicySetCompileErrors alert watches it. When the org is deleted, every node clears it.
ironflow_cron_slots_skipped_total	Counter	environment_id, function_id	Cron schedule slots that produced no run — the engine was down or stalled across the slot, or the fire itself failed after claiming it. Missed slots are never backfilled, so this counter (plus a WARN log carrying the slot count and window) is the only record they were dropped. Non-zero means a scheduled function did not run when it should have. Read the coverage limits below before you rely on it.
ironflow_cancel_on_failures_total	Counter	stage	Store errors while a cancelOn cancel was matched or applied. stage is list (a spec or event read), getrun (the run read), mark (the cancel write). Each error counts, including those a retry then recovered. The engine retries in place for about 350 ms and logs an ERROR if the cancel still did not happen. The reconcile pass then retries it from the stored event.
ironflow_cancel_on_reconciled_total	Counter	—	Runs the cancelOn reconcile pass cancelled from a stored event. Each one is a cancel the live path had not made when the pass ran. Causes include a store error, a cross-node race at run start, an event evicted from the replay buffer, a crash between the event commit and its dispatch, and an ingest backlog that had not reached the event yet. Non-zero is worth a look at the ERROR logs, not an alarm by itself.
ironflow_event_mark_processed_failures_total	Counter	—	Times the processed mark ran out of in-place retries (about 350 ms) after a dispatch. It counts calls, not events: a re-dispatch of the same event can count again. The event stays unprocessed, and the retention sweep keeps its runs, steps and the event itself, unless a later re-dispatch (the files sweeper or an idempotent retry) marks it. The engine logs an ERROR with the event ID and environment. Non-zero means the database failed for longer than the retries.
ironflow_run_result_publish_failures_total	Counter	—	RESULTS frames that failed to publish after the run was already committed, for any terminal or paused status. The run state is correct and the dispatch is not reported as failed. A synchronous caller reads the stored run instead: both InvokeFunctionSync and TriggerSync poll throughout their wait on a 100 ms cadence. Storage latency can delay the read; a lost notification alone does not force the caller to wait until timeout. The engine logs an ERROR (PublishRunResult: publish failed) with the run ID. Usually NATS is unavailable.

Push attempts and step retries

Push duration starts immediately before the HTTP client call and ends after response-body handling. A body-read timeout records timeout even if success headers arrived; other body-read failures record conn_error. A fully read response records its HTTP class, including a 2xx whose body fails JSON decoding. Request encoding and URL construction failures make no HTTP attempt and add no sample.

Redirects followed by the HTTP client use the final response’s outcome. The closed outcome set has no 3xx: an unfollowed redirect records conn_error and its body still follows the existing response decode path. An interrupted request also records conn_error unless its error reports a timeout.

Step retries count only a successfully persisted increase of a step’s attempt number under a newer execution sequence. A streaming start records the retry when it commits; completion and repeated checkpoints do not count it again. Counters reset on process restart. Metrics are best effort: a crash after the database commit and before the increment loses that sample. Streaming also skips a retry sample if the run cannot be read for its identity after the step commit, and logs that at Debug.

Cached database and consumer lag samples

A server-owned sampler refreshes outbox lag, outbox dead-letter counts and active projection rebuilds every 15 seconds. Each collector has a two-second I/O budget. Scrapes read only memory, including pool state. Samples start asynchronously at boot and disappear after a failed refresh; the corresponding *_collector_errors_total counter increments. An absent gauge is not a zero backlog. Values can be up to one sampling interval plus the bounded pass duration old.

NATS lag covers EVENTS, STEPS, RESULTS, PUBSUB and CANCEL. It sums NumPending, excluding messages already delivered and awaiting acknowledgement. Consumer names never become labels. Each NATS refresh has a two-second total budget and a limit of 10,000 consumers per stream. A failed listing, timeout or exceeded limit omits that stream instead of publishing a partial total. A stream with no consumers reports zero.

Every node samples the shared backlog independently. Alerts and Grafana panels use max(), or max by (stream) for NATS, so replicas do not multiply the backlog. Outbox dead-letter panels use sum(max by (environment_id) (...)) to deduplicate nodes before totaling environments.

What cron_slots_skipped_total does and does not cover

A slot counts as missed only when no node claimed it. Evidence is the SYS_cron_triggers dedup bucket, which holds one key per slot some node actually fired and is cluster-wide by construction — so a node coming back from its own outage does not blame itself for slots a healthy peer covered. The WARN log carries a verified field: true means every candidate slot in the window was checked against the bucket, false means the count is a floor (bucket unreachable, lookup budget spent, or the window was clamped).

The bucket’s 24h TTL bounds what is knowable. Consequences, all of which are real:

• Schedules with a period of 24h or more are never reported. A daily or weekly job’s previous claim key has already expired by the time the next slot comes around, so there is no evidence the schedule was ever live and nothing is emitted — even for a 5-minute outage that straddles the slot. Only sub-daily schedules are covered, and only for outages shorter than 24h. For daily and weekly jobs, alert on “did a run happen” (ironflow_runs_total), not on this counter.
• Each node reports independently. A whole-cluster restart has every node compute the same missed set, so the total is inflated by the node count, and a crash-looping node re-reports the same window on every boot until a new slot is claimed. Alert on the condition, not the magnitude.
• Editing a cron expression can inflate the next report once. The dedup key records the slot, not the schedule that produced it, so the window is evaluated against the new expression. Tightening 0 * * * * to */1 * * * * and restarting can invent up to 59 misses per hour of window.
• The most recent slot is held back for two minutes. A slot younger than that may have no key purely because no peer has reached it yet, so counting it would be a false alarm. An engine that comes up inside that window under-reports by at most one slot.

Treat this counter as a detector, not an accountant: non-zero means investigate the WARN logs, which carry the exact window.

Alert on the raw value (ironflow_cron_slots_skipped_total > 0), not on increase(). The downtime increment is written during engine startup, before the first scrape — on a rollout the new pod is a new series whose first sample already carries the count, so increase() has nothing earlier to measure against and reads zero. The bundled CronSlotsSkipped rule latches per instance until that process restarts.

File events and cleanup metrics

Alert on increase(ironflow_file_events_quarantined_total[1h]) > 0. Every node sweeps file events with no claim, so one event can be quarantined and counted once per node. Alert on the condition, not the magnitude.

ironflow_file_orphans_reclaimed_total counts successful background retirement deletion calls. ironflow_file_orphan_bytes_reclaimed_total sums known queued sizes, with zero for unknown sizes. Retries after failed acknowledgement and concurrent workers can count an object again, so these measure approximate work rather than exact storage savings. Immediate API cleanup does not increment them.

To inspect file cleanup, read the cleanup conflict and failure logs, or use files.reconcile: dry-run to inspect queued and unexplained objects. See File storage.

Broker delivery scheduling

ironflow_broker_delivery_deferrals_total{reason="coalesce"} counts duplicate deliveries already queued or active locally. Coalescing is expected during slow processing and does not represent a processing failure. ironflow_broker_delivery_deferrals_total{reason="overflow"} counts deliveries refused by the local scheduling count or byte budget. JetStream retains these payloads for retry. These per-node counters count scheduling decisions, not unique events or confirmed broker acknowledgements. A redelivery after local overflow joins the repeat class even if native processing has not yet started.

Quick Start

# Enable metrics
IRONFLOW_METRICS_ENABLED=true ./ironflow serve

# Verify metrics endpoint
curl http://localhost:9123/metrics

# On a Compose stack, read it on the metrics port
curl http://localhost:9223/metrics

The /metrics endpoint is unauthenticated and exposes identifiers such as function, step, and environment labels. Control access to its listener through your network configuration.

Without a separate port, /metrics stays on the API port. Select a separate listener before you expose that API publicly. The Compose files do this for you: each node serves metrics on its API port plus 100 (9223, 9224, 9225), published only on 127.0.0.1, so the public port answers 404 for /metrics.

Docker Compose with Prometheus

The single-node compose file ships a monitoring profile that starts Prometheus and Grafana alongside Ironflow:

docker compose -f docker-compose.single-node.yml --profile monitoring up

Prometheus will be available at http://localhost:9090 and automatically scrapes Ironflow metrics. Grafana will be available at http://localhost:3000 (anonymous viewer access; admin login admin/admin unless you set GRAFANA_ADMIN_USER and GRAFANA_ADMIN_PASSWORD). Its Ironflow Performance dashboard is provisioned automatically, including a Failures & Resilience row: error rate by function, failures by cause, DLQ writes, outbox dead letters, open circuit breakers, capacity queue age, invoke errors, and projection apply errors.

The Cluster and HA compose files (docker-compose.cluster.yml, docker-compose.ha.yml, and the docker-compose.multi-node.yml development variant) extend this profile with Jaeger for traces and the NATS dashboard. See Docker Compose Deployment → Monitoring.

Grafana Integration

Point Grafana to your Prometheus instance and use these example queries:

# Request rate (per second)
rate(ironflow_http_requests_total[5m])

# Run completion rate by function
rate(ironflow_runs_total{status="completed"}[5m])

# P95 run duration
histogram_quantile(0.95, rate(ironflow_run_duration_seconds_bucket[5m]))

# Active runs
ironflow_active_runs

# Error rate
sum(rate(ironflow_runs_total{status="failed"}[5m])) / sum(rate(ironflow_runs_total[5m]))

# Runs that used up their retries and are waiting for an operator (not in the error rate)
sum by (function_id, environment_id) (increase(ironflow_runs_paused_exhausted_total[1h]))

OpenTelemetry Tracing

When IRONFLOW_OTEL_ENDPOINT is set, Ironflow exports trace data via OTLP gRPC.

Span Hierarchy

For an HTTP-emitted event, the synchronous chain contains the HTTP server span, eventtrigger.Emit, the accepted Run: span, the SDK process, and the executed step.run callback. Each span is a child of the preceding span.

The engine creates one Run: span per push segment or pull assignment. Pull spans end at handoff and measure engine assignment, not worker execution. First HTTP-triggered segments keep the producer parent even after queue delay. Later segments start a new trace linked to the first and preceding segments, with a producer link for an event that resumes a wait. Synchronous child invokes use the current parent segment. Async invokes, debounce, and each event in a batch use links. Each Run span includes ironflow.attempt. The engine creates no step spans. Configured SDKs create process and step.run spans beneath the accepted run context. Add application spans for more detail.

A committed terminal FAILED transition marks the push run span as Error and records a fixed run failed exception with error.type=run_failed. Exception text does not include endpoint errors or user payloads. Retries, pauses, yields, and rejected stale writes do not mark a terminal run failure.

With engine tracing disabled, assignment carriers are empty and accepted span anchors stay unchanged. A configured tracer still propagates valid unsampled spans.

HTTP spans use Error for normalized 5xx responses, including gRPC errors mapped from trailers. Routine stream teardown keeps its existing status.

Each span includes attributes:

• Run spans: run.id, function.id, function.name
• HTTP spans: http.request.method, http.route, url.path, http.response.status_code (derived from Grpc-Status, or stamped by a pre-router rejection, as above)

The HTTP span wraps authentication, so a rejected credential gets a span too. A span for a rejected credential carries ironflow.auth.failure_reason (see Authentication failures). On every request and every status, http.route and url.path both hold the route (for a request refused before the router, the route it was aimed at) or unmatched, never the caller’s path. This deviates from the OpenTelemetry HTTP conventions, where url.path is the raw path: an unauthenticated caller controls that text, and a 200 does not prove the caller authenticated (the dashboard shell answers 200 for /platform/<anything>). Query traces on http.route.

Trace a pull worker with compose Jaeger

Use the runnable Jaeger fixture in the public ironflow-releases repository. It includes compose configuration, a Python polling worker, and a verifier that saves Jaeger JSON and checks span ancestry. Use the engine and SDK from the same release as the fixture.

Run these fish commands. Docker Compose, Python 3.10+ and an installed ironflow binary are required.

git clone --depth 1 https://github.com/sahina/ironflow-releases.git
cd ironflow-releases
python3 -m venv /tmp/ironflow-trace-venv
/tmp/ironflow-trace-venv/bin/pip install 'ironflow-py[tracing]' -r examples/observability/requirements.txt
docker compose -p ironflow-trace-walkthrough -f examples/observability/compose.yaml up -d
set -gx TRACE_STATE (mktemp -d)
env IRONFLOW_OTEL_ENDPOINT=localhost:4317 IRONFLOW_OTEL_SERVICE_NAME=walkthrough-engine IRONFLOW_OTEL_SAMPLE_RATE=1 ironflow serve --dev --host 127.0.0.1 --port 9123 --nats-port -1 --db "$TRACE_STATE/ironflow.db"

This starts an unauthenticated, loopback-only dev engine. Keep it running. Start the worker in another terminal:

env IRONFLOW_SERVER_URL=http://localhost:9123 /tmp/ironflow-trace-venv/bin/python examples/observability/worker.py

In a third terminal, confirm that trace-walkthrough appears in the functions response, then emit and verify:

curl -fsS http://localhost:9123/ironflow.v1.IronflowService/ListFunctions \
  -H 'Content-Type: application/json' -d '{}'
python3 examples/observability/verify.py --output /tmp/ironflow-trace-evidence

The verifier emits an event through HTTP IronflowService/Emit with a W3C parent. It requires one trace containing the HTTP span, first Run: assignment, SDK process, and executed step.run. The worker then sleeps for two seconds and fails its next callback once. Sleep and retry assignments must start separate traces with links to the first and preceding assignments. Memoized callbacks must not create extra step spans. A passing check prints three trace IDs and writes summary.json; failed checks exit nonzero. Export and indexing can take several seconds. Use a new output directory for each run.

Open http://localhost:16686 and search for service walkthrough-engine, with run.id from the emitted result. Open the trace IDs printed by the verifier. The first trace spans both engine and worker services. Later assignment roots link backward to earlier segments. Select their span references to navigate back; use the printed IDs to open later traces if your viewer has no reverse link search. Pull assignment spans end at handoff, so worker children can outlast them. The fixture README describes the exact assertions and coverage limits.

Stop worker and engine with Ctrl-C to flush spans, then stop this Jaeger project. Its in-memory traces are lost on shutdown; retain the exported JSON first.

docker compose -p ironflow-trace-walkthrough -f examples/observability/compose.yaml down

The cluster compose monitoring profile also bundles Jaeger. See Docker Compose Deployment → Monitoring.

Sampling

Control trace sampling with IRONFLOW_OTEL_SAMPLE_RATE:

• 1.0 — Sample all traces (development)
• 0.1 — Sample 10% of traces (production)
• 0.01 — Sample 1% of traces (high-traffic production)

Root spans and requests with a remote parent use the local sampling ratio, whether the incoming sampled bit is set or unset. A remote caller cannot force or suppress recording by flipping that bit. Child spans of a local parent follow that parent’s sampling decision.

---

Instrumenting Your Functions

Ironflow automatically propagates trace context to your functions. Here’s how to use it in your code.

Trace Context Propagation

Pull assignments include context.traceparent and context.tracestate in REST JSON and in streaming JobContext protobuf fields 4 and 5. The carrier identifies the accepted engine assignment span. The poll request and original producer are not substituted for that span. Context is present even when the job has no secrets. Existing protobuf fields 1 through 3, including trace_id, retain their behavior. The Go, Node and Python execution runtimes extract these fields and start a process consumer span. Configure an application OTel provider and exporter to record and export it.

NATS publishers carry W3C traceparent and tracestate headers. The transactional outbox captures the emitter context at enqueue and retains it through dead-letter redrive. Execution entering through NATS starts a new trace linked to that producer. Trace records carry no baggage or event payload, and invalid trace data does not reject an event. tracestate is limited to 512 bytes.

When tracing is enabled, Ironflow injects W3C trace context headers (traceparent, tracestate) into every HTTP request sent to your push-mode functions. This means your function’s spans automatically appear as children of Ironflow’s run span — no manual correlation needed.

Ironflow Server                          Your Function
──────────────                          ──────────────
[Run: order-processor]  ──HTTP POST──▶  [handle-order]
                                           ├── [query-database]
                                           └── [send-notification]

Both sides share the same trace ID, giving you a single end-to-end trace view in Jaeger, Grafana Tempo, or any OTLP-compatible backend.

Push Mode (Next.js, Express, Lambda)

Your function receives traceparent and tracestate as standard HTTP headers alongside Ironflow headers:

Header	Description
traceparent	W3C trace context (trace ID, span ID, flags)
tracestate	Vendor-specific trace data
X-Ironflow-Run-ID	The current run ID
X-Ironflow-Function-ID	The function being executed
X-Ironflow-Attempt	Current retry attempt number

To create child spans in your function, initialize the OTel SDK with W3C propagation and extract the context from incoming headers. Your framework’s OTel instrumentation typically handles this automatically.

Next.js

Set up OTel instrumentation:

instrumentation.ts

import { registerOTel } from "@vercel/otel";

registerOTel({
  serviceName: "my-nextjs-app",
});

Create spans inside your function — they automatically become children of the Ironflow run span:

app/api/ironflow/route.ts

import { serve, createFunction } from "@ironflow/node";
import { trace } from "@opentelemetry/api";

const myFunction = createFunction(
  {
    id: "order-processor",
  },
  async ({ step }) => {
    const tracer = trace.getTracer("my-app");
    const result = await tracer.startActiveSpan("process-order", async (span) => {
      try {
        const order = await step.run("validate", async () => {
          // your logic
        });
        return order;
      } finally {
        span.end();
      }
    });
    return result;
  },
);

export const POST = serve({ functions: [myFunction] });

Express

Initialize the OTel SDK before any other imports:

// tracing.ts (import before anything else)
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";

const sdk = new NodeSDK({
  serviceName: "my-express-app",
  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4317" }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

The Express HTTP instrumentation automatically extracts the traceparent header, so any spans you create inside your function handler are correlated with the Ironflow trace.

Go

Configure your application’s OpenTelemetry tracer provider. The Go SDK continues the engine’s W3C context automatically for push handlers and both worker transports. Use the handler context for additional application spans:

func handle(ctx ironflow.Context) (any, error) {
    goCtx, span := otel.Tracer("orders").Start(ctx.Context(), "validate-order")
    defer span.End()
    return validateOrder(goCtx)
}

The SDK emits a process span per execution and step.run children for callbacks that execute. Replayed steps emit no spans. See the Go SDK reference.

Python

Wrap the ASGI app from serve() in the OTel ASGI middleware (pip install opentelemetry-instrumentation-asgi). It extracts the traceparent header, so spans you create inside a handler are children of the Ironflow run span:

from opentelemetry import trace
from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware

from ironflow.serve import serve
from ironflow.worker import function

tracer = trace.get_tracer("my-service")


@function(id="order-processor")
async def order_processor(ctx):
    with tracer.start_as_current_span("process-order"):
        return await ctx.step.run("validate", validate)  # your logic


app = OpenTelemetryMiddleware(serve([order_processor]))

Pull Mode (Workers)

Polling and ConnectRPC streaming workers continue the accepted assignment’s context. Each execution starts a process consumer span, and an executed step.run callback starts a child step.run span. Memoized steps add no span. A resumed job uses its new assignment context, so its SDK spans join the new linked trace. The poll request’s span does not become the job’s parent.

The SDKs use the OTel API. Your application owns the provider, exporter, sampling, and shutdown flush. Missing or malformed parents start new roots; valid unsampled parents retain their trace identity. Invalid or oversized tracestate is discarded without losing a valid parent. Baggage is not copied. Without a provider, spans are no-ops.

Go

Install the provider and gRPC exporter in your worker’s Go module:

go get go.opentelemetry.io/otel/sdk/trace go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

Configure the global provider before worker.Run(ctx). Keep it alive until the worker has drained, then call its shutdown function:

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/attribute"
)

func setupTracing(ctx context.Context) (func(context.Context) error, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    provider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewSchemaless(
            attribute.String("service.name", "my-go-worker"),
        )),
    )
    otel.SetTracerProvider(provider)
    return provider.Shutdown, nil
}

Use ctx.Context() from an Ironflow handler for application child spans. The SDK scope is github.com/sahina/ironflow-go/ironflow. Callback panic behavior is described in the Go SDK reference.

Node.js

Install the optional API peer and application tracing packages:

pnpm add @opentelemetry/api@^1.9.0 @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-grpc

Save this as tracing.mjs and load it before worker imports with node --import ./tracing.mjs worker.mjs. NodeSDK registers the context manager.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";

export const telemetry = new NodeSDK({
  serviceName: "my-node-worker",
  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4317" }),
});
telemetry.start();

After your worker drains, call await telemetry.shutdown() from its shutdown handler. The SDK scope is @ironflow/node. Without the optional peer, execution works without spans. Step names appear as ironflow.step.name; keep credentials and business data out of step names.

Python

Install the API extra and application provider/exporter:

python3 -m pip install 'ironflow-py[tracing]' opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

Initialize before starting Worker or StreamingWorker:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "my-python-worker"}))
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
))
trace.set_tracer_provider(provider)

Call provider.shutdown() after the worker drains. The SDK scope is ironflow. The runnable fixture includes the complete worker and shutdown path. The tracing extra supplies only the API; the application installs the provider and exporter.

Failures set error status. Durable yields and cancellation do not. SDK tracing omits payloads, secrets, baggage, exception messages, and stack traces. These setup snippets describe the SDK contracts; the supplied live fixture covers Python polling and does not establish Go, Node, or streaming live parity.

Without OTel

If you don’t use OpenTelemetry, the trace headers are harmless — they’re standard HTTP headers that your framework will ignore. You can still use the X-Ironflow-* headers for logging and correlation:

TypeScript

export const POST = serve({
  functions: [
    createFunction(
      { id: "my-function" },
      async ({ event, step, run }) => {
        console.log(`[run=${run.id}] Processing ${event.name}`);
        // ...
      },
    ),
  ],
});

Go

func handler(w http.ResponseWriter, r *http.Request) {
    runID := r.Header.Get("X-Ironflow-Run-ID")
    functionID := r.Header.Get("X-Ironflow-Function-ID")
    log.Printf("[run=%s fn=%s] Processing request", runID, functionID)
    // ...
}

Python

from ironflow.worker import function


@function(id="my-function")
async def my_function(ctx):
    # ctx.logger already carries the run and function IDs.
    ctx.logger.info("[run=%s fn=%s] Processing %s", ctx.run.id, ctx.run.function_id, ctx.event.name)
    # ...

---

How It Works

1. Startup: Ironflow reads observability config from environment variables and from spec.observability in ironflow.yaml
2. Metrics: When enabled, a dedicated Prometheus registry collects metrics from the engine, event publisher, and HTTP middleware
3. Tracing: When configured, the OTel SDK creates a TracerProvider with OTLP export and W3C context propagation
4. Middleware: Every HTTP request gets an automatic trace span and metrics recording
5. Execution: The engine creates a Run: span per segment or pull assignment; configured SDKs create process and executed step.run spans
6. Shutdown: The tracer provider flushes pending spans on graceful shutdown

---

CPU and Memory Profiling

Ironflow supports Go’s pprof profiling via a separate debug listener. Enable it with the --pprof flag:

ironflow serve --pprof

This starts pprof handlers on 127.0.0.1:6060 (localhost only), isolated from the main server’s auth middleware. Capture profiles during load testing:

# Heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# CPU profile (30 seconds)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Goroutine dump
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Compare heap before/after load
go tool pprof -diff_base heap-before.prof heap-after.prof

The make loadtest command captures heap and goroutine profiles automatically before and after each load test run. See Benchmarks for the full workflow.

Health Endpoints

Ironflow provides two health endpoints:

Endpoint	Purpose	Checks	Auth
/health	Liveness and traffic routing	PostgreSQL connectivity	No
/ready	This node can do everything	PostgreSQL, NATS and JetStream connectivity, and every engine stream	No

The /ready endpoint applies a 2-second timeout to its checks. It checks the NATS connection and JetStream for embedded and external NATS alike, including in dev mode. It reports a stream fault as issues["stream:<name>"] from a snapshot the node refreshes every 5 seconds.

Route traffic and restart the engine on /health. Use /ready to alert, to gate a startup order and to check a node by hand. A load balancer that routes on /ready takes every node out for one stuck stream, while the API can still serve. Every Compose node healthcheck reads /health. In Cluster and HA, nginx has its own healthcheck on /ready, which reports the fault and changes no routing. See Recover a stuck JetStream stream.

Structured Logging

The serve command outputs JSON-formatted logs by default for production log pipeline compatibility (Loki, CloudWatch, ELK). All other CLI commands use human-readable console output.

Variable	Description	Default
LOG_FORMAT	Set to text to force human-readable output for serve	JSON for serve, text for CLI
LOG_LEVEL	trace, debug, info, warn, error	info for serve, warn for CLI

SDK user-function logs

The accepted SDK log correlation contract defines five execution fields and explicit bridges to application-owned OTel Logs pipelines. SDK implementation is tracked separately in #3444, #3445, and #3446. This contract does not add platform collection of user-function logs.

Log identity fields

Log queries use run_id, function_id, step_id, and environment_id. Actual environment names use environment_name. Configuration warnings use environment_variable for variable names; CLI environment selection uses environment_selector. These replace ambiguous env, env_id, environment, and camel-case identity log keys. Wire API payload keys are unchanged.

Authentication failures

A rejected credential logs one line at warn, counts once in ironflow_auth_failures_total{reason}, counts in ironflow_http_requests_total under the status it returned (401, 403, or 302 for a dashboard page redirected to the login page with an invalid or expired session cookie), and sets ironflow.auth.failure_reason on its HTTP span. The same applies to REST, ConnectRPC, dashboard page auth, /api/v1/auth/validate, dashboard and platform login, password changes, webhook ingest, and the remote MCP route. The line’s message is Authentication failed, except for env_forbidden, which keeps the message of the scope check that refused it (for example Rejected API key request outside its environment scope). A dashboard page request with no session at all is a first visit, or a scanner probe, and gets a 302 to the login page (or a 401 for a fetch/XHR request); /api/v1/auth/validate with no token, which every logged-out dashboard load asks, gets a 401. None of these is logged or counted as an authentication failure; all still count in the HTTP metrics.

reason	Meaning
no_credential	The request carried no API key, token or session cookie.
unknown_key	The API key matches no stored key.
expired_key	The API key has expired.
expired_token	A JWT has expired.
bad_token	A JWT is malformed, has a bad signature, or has the wrong audience.
revoked_session	The JWT is valid, but its user or key no longer exists or no longer matches.
inactive_user	A platform user is deactivated: a 401 on a token, a 403 at platform login.
bad_login	A dashboard or platform login named an unknown user or gave a wrong password, or a password change gave an incorrect current password.
bad_webhook_token	A webhook ingest token is missing, wrong or names an unknown source.
bad_signature	A webhook signature is missing or does not verify.
wrong_transport	A credential arrived where the route does not take it. The remote MCP route accepts only an Authorization: Bearer header, not a session cookie, ?token= or the WebSocket credential subprotocol. Agent-tools RPCs require an API key and reject dashboard sessions presented as a bearer JWT or session cookie.
env_forbidden	A valid credential asked for an organization or environment outside its scope, returning 403.

The line also carries method, route and remote_addr. It carries principal_id, the API key, user or platform user ID, when the refusal identifies a principal or the request has already authenticated. This includes a known account’s wrong password at login or password change, an org or environment scope violation, a remote MCP transport refusal, and webhook token or signature refusals after API-key authentication. A revoked dashboard or desktop session names its original subject even if that user was deleted. Unknown credentials and requests without a principal carry no principal ID.

In dev mode, authentication is bypassed and the request principal is the synthetic user_admin. Refusal logs identify that dev user. Agent-tools calls that reach the API-key guard report wrong_transport, even if the request presented an API key, because dev mode does not resolve credentials.

route is the matched route pattern; for a rejection before routing, the route the request was aimed at, as the route table resolves it, or unmatched when it names none. It is never the raw path, because a caller can put a pasted key or a megabyte of text there. The line never carries the presented key, token or password, or the login email.

New HTTP authorization audit decisions use this same bounded label in their path payload field. Older audit rows can retain concrete normalized paths under normal retention. Resource IRNs and Rego HTTP inputs keep their actual target values. See Auth audit trail.

The access log (HTTP request) wraps authentication too. It logs at Info with method, bounded route, normalized status, duration, principal_id, environment_id, and request_id. Principal and environment fields are empty when unresolved. Each observed request receives an X-Request-ID response header matching the log. An inbound ID is accepted only when it contains 1 to 128 ASCII letters, digits, dots, underscores, or hyphens; otherwise the server generates one. Request context loggers and completion logs also carry trace_id, span_id, and the boot node_id. Without an active trace, trace and span IDs contain zeros. Authentication supplies the resolved environment_id. /health, /ready, and /metrics completions log at Debug. Stream duration is the connection lifetime; stream metrics still omit latency.

Organization and environment refusals share the env_forbidden warning budget. A flood of either can suppress warnings for the other; counters and span reasons still record every refusal.

Each reason has its own log limit of about 10 lines per second after a burst of 50, so a fleet polling with one revoked key cannot hide a login brute force. The next line that gets through reports how many were dropped in suppressed. An admitted warning includes request_id and permits one paired Info completion log. Refusals whose warning is suppressed log their completion only at Debug. The warning budget stays unchanged; the combined warning and completion budget is at most twice that budget. The counters and the span are not limited. Some refusals of a known principal also write an authz.decision.denied audit row with action auth:authenticate, at most one per principal per minute; a failed platform login never does, even for a known platform user. Refused credentials lists which refusals write one. Authorization denials after a successful login are audited, not logged here.

Grafana Dashboards

Two pre-built Grafana dashboards live in deploy/docker-compose/cluster/grafana/dashboards/:

• ironflow-performance.json — Run throughput, success/failure rates, latency histograms, worker metrics
• nats-monitoring.json — JetStream storage, throughput, consumer lag, slow consumers

The monitoring profile of the Cluster and HA Compose files provisions both into Grafana. The single-node file provisions ironflow-performance only, because it runs no NATS exporter. For your own Grafana, import the JSON files. See Docker Compose Deployment → Monitoring.

Requires IRONFLOW_METRICS_ENABLED=true.

Alerting

Ironflow ships Prometheus alert rules in deploy/docker-compose/monitoring/rules/. The monitoring profile of the single-node Compose file loads the 20 engine rules. The Cluster, multi-node and HA files also load the four NATS rules. The profiles ship no Alertmanager and configure no notification receiver, so an alert shows in Prometheus and notifies nobody. For your own Prometheus, load the rule files and add an Alertmanager. You can also write your own rules from the metrics above. See Docker Compose Deployment.

---

What’s Next?

• Benchmarks — run and interpret performance benchmarks
• Workflows — learn about functions, steps, and execution modes
• Architecture — system design and component overview
• Configuration — full environment variable reference

Metric label migration before 1.0

Update external PromQL, recording rules, alert annotations and dashboard variables when upgrading: environment becomes environment_id, function becomes function_id, and projection name and env_id become projection and environment_id. The sampled projection gauge renames env to environment_name; push-side projection counters and histograms drop the old env label because it was guessed from an ID suffix rather than read from an authoritative name source. The outbox dead-letter gauge’s env held an ID and becomes environment_id. Step counters and durations now include environment_id. Reimport the shipped dashboards. No legacy series are emitted; historical data retains its previous labels. Split queries at the upgrade boundary when comparing history.

This non-exception break resets ADR 0125’s clean window. Coordinate the actual breaking release with #2983; C4 requires at least fourteen days and two clean releases after it.

Temporarily change the log level

An admin of the default organization or a platform admin can call the authenticated API to change the process log level without restarting. The route requires cluster:log-level and is unavailable on the separate metrics listener.

curl -X POST "$IRONFLOW_URL/api/v1/cluster/log-level" \
  -H "Authorization: Bearer $IRONFLOW_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"level":"debug","ttl":"15m"}'

Use trace, debug, info, warn, or error. Omitting ttl selects 15 minutes; the maximum is 4 hours. Every accepted attempt writes a platform audit event, cluster.log_level.requested, before changing the level. If that audit write fails, the request fails without applying or broadcasting the override.

Multi-node clusters require the same explicit auth.jwtSecret or IRONFLOW_JWT_SECRET on every node. Auto-generated per-node keys do not meet this prerequisite. The command and each acknowledgement are signed with the current JWT key. Rotate shared keys consistently across the cluster.

The response’s applied_node_ids contains the sorted node IDs that acknowledged this command, including the initiating node if it applied it. delivery: "best_effort" means cluster fanout with a bounded one-second acknowledgement window. It does not mean every node changed. A missing ID can indicate an unavailable node, a different signing key, a rejected timestamp, or a lost reply. delivery: "publish_failed" means local application may have happened but publication failed. delivery: "local" means a single-node operation.

Overrides expire at the absolute expires_at time and restore the boot level. A newer command replaces the previous timer. Keep node clocks synchronized. Fanout is ephemeral: disconnected nodes miss it, and restarting a node restores its boot baseline. A response can arrive after a short override expires or after another operator supersedes it. No override survives a restart.