Observability

Ironflow provides built-in observability through OpenTelemetry distributed tracing and Prometheus metrics. Both are optional — zero overhead when disabled.

Key Concepts

Feature	Description
Prometheus Metrics	Counter, histogram, and gauge metrics for runs, steps, events, and HTTP requests
OpenTelemetry Tracing	Distributed trace spans for workflow runs, steps, and HTTP requests
W3C Trace Context	Automatic propagation of `traceparent`/`tracestate` headers
Zero Overhead	No performance impact when observability is disabled

Configuration

Configure observability via environment variables:

Variable	Description	Default
`IRONFLOW_METRICS_ENABLED`	Enable Prometheus metrics at `/metrics`	`false`
`IRONFLOW_OTEL_ENDPOINT`	OTLP gRPC endpoint for tracing (empty = disabled)	(empty)
`IRONFLOW_OTEL_SAMPLE_RATE`	Trace sampling rate (0.0 to 1.0)	`1.0`
`IRONFLOW_OTEL_SERVICE_NAME`	Service name in trace data	`ironflow`
`IRONFLOW_OTEL_INSECURE`	Use plaintext gRPC for OTLP export (set `false` for TLS)	`true`

Every one of these has a spec.observability equivalent in ironflow.yaml. A field set in the file wins over the environment variable; a field the file leaves out keeps its environment value.

Prometheus Metrics

When IRONFLOW_METRICS_ENABLED=true, the /metrics endpoint serves Prometheus exposition format on the main API port.

Core Metrics

The registry exposes 45 metric families. The 19 below are the ones you reach for day to day, and they include every metric a bundled alert rule queries. The other 26 are subsystem-internal; scrape /metrics to see them in full. By family: capacity and dispatch (ironflow_capacity_*, 9 more beyond the two below), blob overflow (ironflow_blob_*, 5), projection rebuild (ironflow_projection_rebuild_*, 4), engine invoke (ironflow_engine_* and ironflow_invoke_dispatch_drained_total, 4), plus ironflow_policies_rejected_total, ironflow_authz_cross_tenant_denied_total, ironflow_event_schema_checks_total and ironflow_system_event_drops_total.

Metric	Type	Labels	Description
`ironflow_runs_total`	Counter	`function_id`, `status`, `environment`, `failure_cause`	Total workflow runs. `failure_cause` is `platform` or `user` for failed runs, empty for completed.
`ironflow_run_duration_seconds`	Histogram	`function_id`, `environment`	Run execution duration
`ironflow_steps_total`	Counter	`function_id`, `status`	Total step executions. Not labeled by step ID: step IDs are yours to choose and are commonly interpolated with request data (step.run(`search-${query}`)), which would make the label unbounded. Use the runs API or your traces for a per-step breakdown.
`ironflow_step_duration_seconds`	Histogram	`function_id`	Step execution duration. Not labeled by step ID, as above.
`ironflow_events_emitted_total`	Counter	`event_name`, `environment`	Events published to the pub/sub stream: your `emit()` events under their own name (counted when the outbox publishes them), and engine lifecycle frames (`run.created`, `step.completed`, …) under `run.` / `step.`. Filter on `event_name` to separate the two. Event names are trigger-matched identifiers — keep them static; a name interpolated with request data (emit(`order.${id}`)) adds one series per value.
`ironflow_active_runs`	Gauge	`function_id`, `environment`	Currently active runs
`ironflow_http_requests_total`	Counter	`method`, `path`, `status_code`	HTTP requests. `path` is the matched route pattern (`/api/v1/events/{id}`), or `unmatched` when no route matched — never the raw request path, so cardinality is bounded by the route table. A request whose handler panics is counted with `status_code="500"`. For gRPC and gRPC-Web the label is not the wire status line: those protocols answer 200 and put the error in trailers, so `status_code` is derived from `Grpc-Status` and matches what the same error returns over Connect — a shed write is 429 whichever protocol asked. A request rejected before the router runs (a throttled write) carries no readable status on every protocol, so the rejecting middleware stamps its code instead. On a gRPC or gRPC-Web stream, `Canceled` and `DeadlineExceeded` stay 200 — that is a client disconnecting or timing out, not a failure — while every other code counts, so a stream that dies of `Internal` is not recorded as a success. A Connect-protocol stream is the exception: it carries no `Grpc-Status` to read, so its errors still count as 200.
`ironflow_http_request_duration_seconds`	Histogram	`method`, `path`	HTTP request duration. Same `path` label as above.
`ironflow_workers_connected`	Gauge	—	Currently connected pull-mode workers
`ironflow_worker_disconnects_total`	Counter	`reason`	Worker disconnections by reason
`ironflow_outbox_dead_letter_count`	Gauge	`env`	Current rows in the outbox dead-letter table per environment (computed at scrape time — zero drift across nodes and restarts). See the Outbox DLQ runbook.
`ironflow_outbox_dlq_collector_errors_total`	Counter	—	Scrape-time failures of the DLQ count query. Non-zero means the gauge above is stale; distinguishes “DLQ is empty” from “collector can’t query the DB”.
`ironflow_dlq_writes_total`	Counter	`source`	DLQ moves since process start. `source="outbox"` fires every time the outbox worker exhausts the retry budget (default 10) and moves a row to `outbox_dead_letter`.
`ironflow_circuit_breaker_state`	Gauge	`function_id`, `state`	Breaker state by function: `0`=closed, `1`=open, `2`=half-open. An open breaker blocks dispatch to that function.
`ironflow_codec_encode_errors_total`	Counter	`reason`	SubjectCodec encode failures (`invalid_namespace`, `empty_topic`, `length_exceeded`, `wildcard_in_user_input`, `panic`).
`ironflow_codec_decode_errors_total`	Counter	`namespace`, `reason`	SubjectCodec decode failures (`malformed`, `unknown_namespace`, `empty_body`, `panic`). Silent message drop on the projection consumer path — projections lag.
`ironflow_capacity_leases_expired_total`	Counter	`environment`, `function_id`	Concurrency leases expired and reclaimed by the scanner.
`ironflow_capacity_oldest_queue_age_seconds`	Gauge	`environment`, `function_id`	Age of the oldest waiting `dispatch_queue` row. Global snapshot — aggregate with `max by(environment, function_id)`.
`ironflow_cron_slots_skipped_total`	Counter	`environment`, `function_id`	Cron schedule slots that produced no run — the engine was down or stalled across the slot, or the fire itself failed after claiming it. Missed slots are never backfilled, so this counter (plus a `WARN` log carrying the slot count and window) is the only record they were dropped. Non-zero means a scheduled function did not run when it should have. Read the coverage limits below before you rely on it.

What `cron_slots_skipped_total` does and does not cover

A slot counts as missed only when no node claimed it. Evidence is the SYS_cron_triggers dedup bucket, which holds one key per slot some node actually fired and is cluster-wide by construction — so a node coming back from its own outage does not blame itself for slots a healthy peer covered. The WARN log carries a verified field: true means every candidate slot in the window was checked against the bucket, false means the count is a floor (bucket unreachable, lookup budget spent, or the window was clamped).

The bucket’s 24h TTL bounds what is knowable. Consequences, all of which are real:

Schedules with a period of 24h or more are never reported. A daily or weekly job’s previous claim key has already expired by the time the next slot comes around, so there is no evidence the schedule was ever live and nothing is emitted — even for a 5-minute outage that straddles the slot. Only sub-daily schedules are covered, and only for outages shorter than 24h. For daily and weekly jobs, alert on “did a run happen” (ironflow_runs_total), not on this counter.
Each node reports independently. A whole-cluster restart has every node compute the same missed set, so the total is inflated by the node count, and a crash-looping node re-reports the same window on every boot until a new slot is claimed. Alert on the condition, not the magnitude.
Editing a cron expression can inflate the next report once. The dedup key records the slot, not the schedule that produced it, so the window is evaluated against the new expression. Tightening 0 * * * * to */1 * * * * and restarting can invent up to 59 misses per hour of window.
The most recent slot is held back for two minutes. A slot younger than that may have no key purely because no peer has reached it yet, so counting it would be a false alarm. An engine that comes up inside that window under-reports by at most one slot.

Treat this counter as a detector, not an accountant: non-zero means investigate the WARN logs, which carry the exact window.

Alert on the raw value (ironflow_cron_slots_skipped_total > 0), not on increase(). The downtime increment is written during engine startup, before the first scrape — on a rollout the new pod is a new series whose first sample already carries the count, so increase() has nothing earlier to measure against and reads zero. The bundled CronSlotsSkipped rule latches per instance until that process restarts.

Quick Start

# Enable metrics
IRONFLOW_METRICS_ENABLED=true ./ironflow serve

# Verify metrics endpoint
curl http://localhost:9123/metrics

The /metrics endpoint does not require authentication — Prometheus has no way to carry an API key, so the endpoint is meant to be reached from inside the cluster (the Helm chart’s ServiceMonitor scrapes the Service directly) and gated at the edge for anything outside it. Treat its payload as sensitive: it exposes every function_id, step_name and environment ID, plus normalized request paths.

If you expose the API through an Ingress, block /metrics there. The bundled chart routes / Prefix, so ingress.blockMetrics (default true) renders a deny rule — enforced for ingress-nginx, and surfaced as a NOTES warning for any other controller, where blocking the path is your edge’s job.

Even on ingress-nginx the rule is not self-enforcing: the controller’s allow-snippet-annotations defaults to false since v1.9, and with it off the snippet is silently dropped (or the admission webhook rejects the Ingress). Turn it on, or block /metrics at your edge — then curl https://<host>/metrics and confirm a 403.

Docker Compose with Prometheus

The single-node compose file ships a monitoring profile that starts Prometheus alongside Ironflow:

docker compose -f docker-compose.single-node.yml --profile monitoring up

Prometheus will be available at http://localhost:9090 and automatically scrapes Ironflow metrics.

The cluster compose files (docker-compose.cluster.yml, docker-compose.multi-node.yml) extend this profile into a full local stack — Prometheus, Jaeger for traces, and Grafana with the datasource and dashboards auto-provisioned. See Docker Compose Deployment → Monitoring.

Grafana Integration

Point Grafana to your Prometheus instance and use these example queries:

# Request rate (per second)
rate(ironflow_http_requests_total[5m])

# Run completion rate by function
rate(ironflow_runs_total{status="completed"}[5m])

# P95 run duration
histogram_quantile(0.95, rate(ironflow_run_duration_seconds_bucket[5m]))

# Active runs
ironflow_active_runs

# Error rate
rate(ironflow_runs_total{status="failed"}[5m]) / rate(ironflow_runs_total[5m])

OpenTelemetry Tracing

When IRONFLOW_OTEL_ENDPOINT is set, Ironflow exports trace data via OTLP gRPC.

Span Hierarchy

HTTP Request (server span)
  └── Run: {functionID} (internal span)
       ├── Step: {stepName} (internal span)
       ├── Step: {stepName} (internal span)
       └── Step: {stepName} (internal span)

Each span includes attributes:

Run spans: run.id, function.id, function.name
Step spans: step.id, step.name, step.type, run.id
HTTP spans: http.request.method, url.path, http.response.status_code (derived from Grpc-Status, or stamped by a pre-router rejection, as above)

Quick Start with Jaeger

# Start Jaeger (all-in-one)
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/jaeger:latest

# Start Ironflow with tracing
IRONFLOW_OTEL_ENDPOINT=localhost:4317 ./ironflow serve

View traces at http://localhost:16686.

The cluster compose files bundle this same Jaeger all-in-one under their monitoring profile, so for a multi-node cluster you don’t need to run it by hand — see Docker Compose Deployment → Monitoring.

Sampling

Control trace sampling with IRONFLOW_OTEL_SAMPLE_RATE:

1.0 — Sample all traces (development)
0.1 — Sample 10% of traces (production)
0.01 — Sample 1% of traces (high-traffic production)

Parent-based sampling is used: if an incoming request carries a sampled trace context, it will always be sampled regardless of the rate.

Instrumenting Your Functions

Ironflow automatically propagates trace context to your functions. Here’s how to use it in your code.

Trace Context Propagation

When tracing is enabled, Ironflow injects W3C trace context headers (traceparent, tracestate) into every HTTP request sent to your push-mode functions. This means your function’s spans automatically appear as children of Ironflow’s run span — no manual correlation needed.

Ironflow Server                          Your Function
──────────────                          ──────────────
[Run: order-processor]  ──HTTP POST──▶  [handle-order]
  ├── [Step: validate]                    ├── [query-database]
  ├── [Step: charge]                      └── [send-notification]
  └── [Step: fulfill]

Both sides share the same trace ID, giving you a single end-to-end trace view in Jaeger, Grafana Tempo, or any OTLP-compatible backend.

Push Mode (Next.js, Express, Lambda)

Your function receives traceparent and tracestate as standard HTTP headers alongside Ironflow headers:

Header	Description
`traceparent`	W3C trace context (trace ID, span ID, flags)
`tracestate`	Vendor-specific trace data
`X-Ironflow-Run-ID`	The current run ID
`X-Ironflow-Function-ID`	The function being executed
`X-Ironflow-Attempt`	Current retry attempt number

To create child spans in your function, initialize the OTel SDK with W3C propagation and extract the context from incoming headers. Your framework’s OTel instrumentation typically handles this automatically.

Set up OTel instrumentation:

import { registerOTel } from "@vercel/otel";

registerOTel({
  serviceName: "my-nextjs-app",
});

Create spans inside your function — they automatically become children of the Ironflow run span:

import { serve, createFunction } from "@ironflow/node";
import { trace } from "@opentelemetry/api";

const myFunction = createFunction(
  {
    id: "order-processor",
  },
  async ({ step }) => {
    const tracer = trace.getTracer("my-app");
    const result = await tracer.startActiveSpan("process-order", async (span) => {
      try {
        const order = await step.run("validate", async () => {
          // your logic
        });
        return order;
      } finally {
        span.end();
      }
    });
    return result;
  },
);

export const POST = serve({ functions: [myFunction] });

Initialize the OTel SDK before any other imports:

// tracing.ts (import before anything else)
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";

const sdk = new NodeSDK({
  serviceName: "my-express-app",
  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4317" }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

The Express HTTP instrumentation automatically extracts the traceparent header, so any spans you create inside your function handler are correlated with the Ironflow trace.

Extract the trace context from Ironflow’s headers and create child spans:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Extract trace context from Ironflow's headers
    propagator := otel.GetTextMapPropagator()
    ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))

    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "handle-order")
    defer span.End()

    // Your logic — this span is a child of the Ironflow run span
}

Pull Mode (Workers)

Pull-mode workers poll the server over HTTP (a ConnectRPC streaming path exists but polling is the wired default). Trace context propagation for pull mode is planned for a future release. In the meantime, you can manually correlate traces using the run ID and function ID from the job assignment.

Without OTel

If you don’t use OpenTelemetry, the trace headers are harmless — they’re standard HTTP headers that your framework will ignore. You can still use the X-Ironflow-* headers for logging and correlation:

TypeScript
Go

export const POST = serve({
  functions: [
    createFunction(
      { id: "my-function" },
      async ({ event, step, run }) => {
        console.log(`[run=${run.id}] Processing ${event.name}`);
        // ...
      },
    ),
  ],
});

func handler(w http.ResponseWriter, r *http.Request) {
    runID := r.Header.Get("X-Ironflow-Run-ID")
    functionID := r.Header.Get("X-Ironflow-Function-ID")
    log.Printf("[run=%s fn=%s] Processing request", runID, functionID)
    // ...
}

How It Works

Startup: Ironflow reads observability config from environment variables
Metrics: When enabled, a dedicated Prometheus registry collects metrics from the engine, step manager, event publisher, and HTTP middleware
Tracing: When configured, the OTel SDK creates a TracerProvider with OTLP export and W3C context propagation
Middleware: Every HTTP request gets an automatic trace span and metrics recording
Engine: Run execution creates parent spans; step execution creates child spans
Shutdown: The tracer provider flushes pending spans on graceful shutdown

CPU and Memory Profiling

Ironflow supports Go’s pprof profiling via a separate debug listener. Enable it with the --pprof flag:

ironflow serve --pprof

This starts pprof handlers on 127.0.0.1:6060 (localhost only), isolated from the main server’s auth middleware. Capture profiles during load testing:

# Heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# CPU profile (30 seconds)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Goroutine dump
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Compare heap before/after load
go tool pprof -diff_base heap-before.prof heap-after.prof

The make loadtest command captures heap and goroutine profiles automatically before and after each load test run. See Benchmarks for the full workflow.

Health Endpoints

Ironflow provides two health endpoints for Kubernetes probes:

Endpoint	Purpose	Checks	Auth
`/health`	Liveness probe	PostgreSQL connectivity	No
`/ready`	Readiness probe	PostgreSQL + NATS connectivity	No

The /ready endpoint uses a 2-second timeout on database checks and skips the NATS check when running in dev mode (embedded NATS). In production Helm deployments, the readiness probe points to /ready and the liveness probe points to /health. This prevents NATS transient blips from cascading into pod restarts.

Structured Logging

The serve command outputs JSON-formatted logs by default for production log pipeline compatibility (Loki, CloudWatch, ELK). All other CLI commands use human-readable console output.

Variable	Description	Default
`LOG_FORMAT`	Set to `text` to force human-readable output for `serve`	JSON for serve, text for CLI
`LOG_LEVEL`	`trace`, `debug`, `info`, `warn`, `error`	`info` for serve, `warn` for CLI

Grafana Dashboards

Four pre-built Grafana dashboards are included in the Helm chart at deploy/helm/ironflow/dashboards/:

ironflow-performance.json — Run throughput, success/failure rates, latency histograms, worker metrics
k8s-infrastructure.json — Pod CPU/memory, node status, volume usage, restart rates
nats-monitoring.json — JetStream storage, throughput, consumer lag, slow consumers
postgres-cnpg.json — Connection count, query performance, cache hit ratio, replication lag

Dashboards are deployed as Kubernetes ConfigMaps when monitoring.dashboards.enabled=true. Grafana auto-imports them via sidecar (label grafana_dashboard: "1"). For standalone Grafana, import the raw JSON files directly.

The cluster Docker Compose monitoring profile auto-provisions the two compose-relevant dashboards (ironflow-performance, nats-monitoring) into a Grafana instance — the k8s-infrastructure and postgres-cnpg dashboards are Kubernetes/CNPG-specific and apply only to the Helm deployment. See Docker Compose Deployment → Monitoring.

Requires IRONFLOW_METRICS_ENABLED=true.

Production Monitoring Stack

Ironflow’s monitoring stack is deployed via the Helm chart and CLI prerequisites. Set monitoring.dashboards.enabled=true and monitoring.alerts.enabled=true in your values file (enabled by default in small/medium/large/multi-tenant templates).

Components:

kube-prometheus-stack (CLI prerequisite) — Prometheus, Grafana, Alertmanager, kube-state-metrics
BlackBox Exporter (bootstrap.sh only) — Synthetic HTTP/TCP probes for /health, NATS, and PostgreSQL
Healthchecks.io — External dead man’s switch (detects total cluster failure)

Alerts: 19 PrometheusRule alerts across four groups — ironflow.critical (7 rules), ironflow.warning (4 rules), ironflow.sre (6 rules with mixed severities), and ironflow.capacity (2 rules) — covering pod health, error rates, latency, NATS/PG connectivity, disk space, probe and Alertmanager failures, memory pressure, worker status, circuit breakers, DLQ backlog, codec encode/decode drops, missed cron slots, and capacity lease expiry and queue age. Deployed by the Helm chart via templates/ironflow-alerts.yaml.

Additional alerts: NATS-specific alerts (templates/nats-alerts.yaml, 4 rules) and PostgreSQL alerts (templates/pg-alerts.yaml, 8 rules) are also in the Helm chart.

Deployment: ironflow deploy --template medium --name staging installs kube-prometheus-stack as a prerequisite and deploys dashboards + alerts via Helm. The Hetzner bootstrap script (deploy/helm/bootstrap.sh) also supports the full stack; pass --skip-monitoring to that script to deploy without it. ironflow deploy has no such flag.

What’s Next?

Benchmarks — run and interpret performance benchmarks
Workflows — learn about functions, steps, and execution modes
Architecture — system design and component overview
Configuration — full environment variable reference