Benchmarks

Ironflow ships a benchmark suite that measures component throughput and end-to-end latency. Use it to establish baselines for your hardware and detect regressions.

Quick Start

Every command runs on embedded SQLite and then on PostgreSQL, and takes a target so one benchmark can be run at a time. Run a target without arguments to list the choices.

# Component benchmarks (in-process, no server needed)
make bench BENCH=Store_CreateRun

# One load-test scenario (starts server, runs k6, captures pprof)
make loadtest SCENARIO=order-saga

# Did a change help? Base commit vs this tree, alternating runs, benchstat report
make bench-ab BENCH=Store_CreateRun          # BASE=<merge-base with main> RUNS=6 by default
make loadtest-ab SCENARIO=order-saga

# Everything, both backends (well over an hour)
make bench-all

# Benchmark and load-test regression checks without a sustained load run
make test-benchmark-harness
make test-loadtest-functions   # bench worker helpers; builds the JS SDK first

Prerequisites

Tool	Required for	Install
Go 1.27+	make bench	Already required
Node.js and pnpm	make loadtest, make loadtest-ab	Already required, because the server embeds the dashboard
k6	make loadtest	brew install k6
Python 3	Load-test baseline comparison and regression checks	Already required
Docker	make bench, make loadtest (the PostgreSQL half)	Already required for PostgreSQL tests

Each benchmark or scenario gets a fresh SQLite database file, or a disposable database in the repository’s PostgreSQL test container. A failure on the first backend still runs the second.

To run one backend, set LIVE_DB on the underlying command: env LIVE_DB=postgres go test -run '^$' -bench . ./tests/benchmark/ (after make test-pg-up) or env LIVE_DB=postgres tests/loadtest/run.sh. The load-test runner rejects IRONFLOW_DATABASE_URL; unset it before running load tests. Application database URLs are never used. The runner drops its databases after each scenario and stops the container if it started it and no other live test databases remain.

The separate PostgreSQL outbox benchmarks also use disposable databases in that container:

make test-pg-up
env LIVE_DB=postgres go test ./internal/store -run '^$' -bench 'Benchmark(EnqueueOutboxPostgres|InsertOutboxInPgTx)' -benchmem

Component Benchmarks (make bench)

Runs the Go testing.B benchmarks matching BENCH with embedded NATS, first against a SQLite database file (embedded mode), then against PostgreSQL. No running server needed. Each pass is saved to build/bench-sqlite.txt and build/bench-postgres.txt. Benchmarks that do not touch the store (NATS, Pattern, Pub/sub ack cost) report the same work on both passes.

What’s measured:

Category	Benchmarks
Store	CreateRun, CreateRun (parallel), GetRun, CreateStep, CreateEvent, ListRuns, StepHistoryLookup, CreateAndCompleteStep
Engine	PushReplay with 1, 10, and 100 completed steps. SyncPolicyRun: a push run of 1 or 10 durable steps on JetStream file storage, under sync=always and sync=2m (ADR 0134)
NATS	Publish, KV Put, KV Get, PublishSubscribe, SyncPolicy (sync=always against sync=2m)
Outbox	SyncPolicyDrain: events per second through the outbox worker into JetStream file storage, under both sync policies
Pattern	Parse, Match, MatchWildcard, PatternMatcher
API	EmitEvent, GetRun, GetRunAPIKey (GetRun through API-key auth), ListEvents, HealthCheck
Pub/sub ack cost	PubSubAckCost with BringUp, Delivery, and Reap: one shared no-ack consumer against one explicit-ack consumer for each subscriber

TestGoroutineLeak dispatches real push functions and verifies their persisted durable steps before checking goroutine growth. TestBootTime starts ten isolated server processes and reports median and p95 time to a healthy response. A failed start fails the test and includes child-process diagnostics. A median above 500 ms logs a warning. Neither runs under make bench; run them with make embed build && go test -run 'TestBootTime|TestGoroutineLeak' ./tests/benchmark/. Boot measurement skips if the executable is missing.

The storage benchmarks measure direct database operations. PushReplay exercises the push executor’s request and completed-step replay path. Setup is excluded from timing. Go benchmarks report time and allocations without enforcing a general speed threshold.

Reading the Output

BenchmarkStore_CreateRun-10     85423    14025 ns/op    2048 B/op    42 allocs/op
│                         │       │        │              │             │
│                         │       │        │              │             └ heap allocations per op
│                         │       │        │              └ bytes allocated per op
│                         │       │        └ nanoseconds per operation
│                         │       └ iterations run
│                         └ GOMAXPROCS
└ benchmark name

Lower ns/op = faster. Lower B/op and allocs/op = less GC pressure.

Comparing a Change (make bench-ab, make loadtest-ab)

To show that a change makes something faster, compare it with a base commit on the same machine in the same session, not with numbers recorded earlier. scripts/bench-ab.sh does this:

1. Builds the base commit (BASE, default: the commit where this branch left main, so the comparison measures only this branch) in a temporary worktree and builds this tree. Only the code under test differs: the base tree gets this tree’s tests/benchmark/, and both load-test arms share this tree’s k6 scripts, SDK worker, and dashboard.
2. Runs RUNS rounds (default 6) on SQLite and on PostgreSQL. Each round runs both arms and swaps their order, so drift on the machine reaches both arms.
3. Prints a benchstat report per backend. Load-test runs become benchmark lines (BenchmarkK6/<scenario>/<metric>, one unit per percentile), so both kinds get the same significance test. A ~ in the delta column means no significant difference.

The script refuses to start when the load average is above half the CPU count; set BENCH_FORCE=1 to override. Results and logs stay in build/bench-ab/<timestamp>/.

Load Tests (make loadtest)

The load tests run five scenarios. Each scenario models one kind of Ironflow application. Every scenario runs on embedded SQLite and then on PostgreSQL, with the same script, the same rate, and the same limits on both.

For each scenario, run.sh starts a fresh authenticated Ironflow server, an SDK worker, and a new database. It does not start the server with --dev. Each scenario has its own database, NATS storage, and bootstrap key. Profiling uses the server’s fixed port 6060. A failed or empty profile capture fails the run.

How a scenario measures

• Open model. Every scenario sends work at a fixed arrival rate (constant-arrival-rate, or ramping-arrival-rate for webhook-burst). The load does not drop when the server slows down. A scenario that cannot start an iteration in time counts it as a dropped iteration. Any dropped iteration fails the run.
• Server timestamps. Each trigger event carries sentAt, the k6 clock at send time. k6 and the server share the host clock. For order-saga, durable-steps, and agent-loop, the latency is the run endedAt minus sentAt. The scenario subtracts the time it adds on purpose, such as an approval delay or a simulated model call. The result is the time the engine adds. webhook-burst and ledger-contention use other sources, which the scenario table names.
• Warm-up. For all scenarios except webhook-burst, runs that start in the first WARMUP seconds (default 30 s) do not count in the latency metrics. The correctness audit still checks them. webhook-burst has no warm-up. Its first phase runs at the low base rate.
• Correctness audit. After the load, each scenario waits for its open runs to finish, up to a time limit. Then it reads back what it started and checks the rules in the scenario table. ledger-contention reads streams and projections. order-saga reads runs, and also entity streams and its projection. The other scenarios read runs. A run that loses work, repeats work, or leaves work open fails. When runs are still open at the limit, the run fails on that alone: the checks that need every run finished print “not checked, run not drained” instead of a second failure, and the webhook-burst drain time is not recorded.
• Sample floors. The main latency metrics each have a companion counter with a minimum count. A metric that collects no samples fails. It does not pass as 0. One metric has no floor: the webhook-burst drain time. The emit latency during the webhook-burst spike has a floor on the number of spike emits. The order-saga cancel time has a floor when CANCEL_RATE is above 0.

Scenarios

Scenario	What it models	What it measures	Correctness audit
order-saga	One order from placement to payment. It writes streams, waits for approval, runs two payment steps, updates a projection, and sends a notification. 5% of orders are cancelled.	Engine time from order placed to order paid. It excludes the approval wait and the simulated gateway calls. Also the time to cancel.	Completed plus cancelled runs equal all runs. A sample of paid order streams (at most 200) has exactly three events each. SOAK=1 skips this sample. The projection count equals the paid count. The notifications received equal the completed count.
durable-steps	A function with five steps and a one second durable sleep.	Gated: the time the engine adds to a six-step run, without the sleep wake-up delay (ironflow_steps_engine), and how late the sleep ends (ironflow_sleep_wake). Reported, not gated: the total time the engine adds, with the delay (ironflow_steps_overhead). The sleep scheduler checks for due sleeps on a fixed interval, so the sleep ends up to one interval late.	Every run completes. No step runs twice.
agent-loop	An AI agent run. It has ten steps that each wait 200 ms (a stubbed model call), and a human approval wait after the fifth step. Many runs are open at once.	Time the engine adds, without the step time and the human wait. Also the peak number of open runs.	Every run completes. The ten step results are in order. No step runs twice.
webhook-burst	Bursty inbound webhooks. Each event starts three functions. One function allows 2 concurrent runs for each customer. The rate goes from a low base to a short spike and back.	Emit request latency during the spike. Also the drain time: how long after the spike ends until all runs sent before that time are complete.	Exactly three runs for each event. No customer has more than two capped runs at the same time.
ledger-contention	Event-sourced accounts. Most deposits go to a few hot accounts, so appends race. A failed append reads the stream head and retries. A small share of iterations (2% by default) send a forced pair: two appends to one dedicated pair account with the same expected version, at the same time. Normal deposits never use the pair accounts. A managed projection folds the deposits.	Duration of the append request that succeeds. Also the natural conflict rate, the retry-exhausted rate, and the projection lag. The forced pairs are a separate check: each pair must give one success and one conflict.	For each account, the projection balance and the projection version equal the stream head.

The scenario scripts are in tests/loadtest/scripts/. The workflow functions are in tests/loadtest/functions/. They all run on one Node worker.

Pass and fail rules

Each scenario has a service level objective (SLO) in tests/loadtest/slo.json. It has a default rate, and one or more metrics with a percentile and a limit. The limit is a k6 threshold. The same limit applies on both backends.

A run passes only when all of these are true:

• No iteration is dropped.
• Fewer than 1% of HTTP requests fail.
• Each SLO metric stays inside its limit, when it has a limit. A metric without a limit has no latency gate.
• The audit counts zero violations and finishes.
• No run is in a failed status, and the outbox has no dead letters.
• No step runs twice. run.sh reads the worker logs for this check.
• No emit that must start a run returns without it. The scenario checks the run IDs in each emit response.
• For order-saga, the notifications received equal the completed runs.
• For ledger-contention, fewer than 1% of deposits use up all retries. Each forced pair gives exactly one success and one conflict. Two successes mean the stream accepted two appends for one version, and the run fails. Normal deposits cannot move a pair account, so the host speed does not change this result. If both appends of a pair get a conflict, another pair moved the stream head first. The scenario then reads the head again and sends the pair again, up to 3 more times. The run must also send enough pairs (at least a quarter of the expected number). The natural conflict rate is reported but has no gate, because it depends on the speed of the host. Set FORCED_PAIR_SHARE to change the share of forced pairs. It must be a number above 0 and at most 1. The run stops at start with an error for 0, because a run with no pair fails the pair count.

The runner also labels a run “worker-bound” when the p99 delay of the worker event loop is above 50 ms. The run still passes or fails by the rules above. The label tells you that the worker, not the engine, may limit the numbers. Set WORKER_PROCESSES=N to start more worker processes.

Run options

Run one scenario on both backends, or all of them:

make loadtest SCENARIO=order-saga
make loadtest SCENARIO=all

These variables change a run. Pass them on the command line.

Variable	Effect
RATE, DURATION, WARMUP	Arrival rate, measured window (default 3m), and warm-up (default 30s). DURATION and WARMUP need a unit: 30s, 2m or 1h. A plain number is an error. webhook-burst has a fixed ramp, so only MAX_VUS applies to it.
MAX_VUS	Upper limit of k6 virtual users.
WORKER_CONCURRENCY, WORKER_PROCESSES	In-flight jobs per worker process, and the number of worker processes. Keep the concurrency above the number of runs that execute a step at the same time.
NATS_SYNC_INTERVAL	JetStream fsync policy of the server the runner boots: always or a Go duration such as 2m (ADR 0134). Unset keeps the server default, an fsync on every write. A duration can lose acknowledged writes on power loss, so the runner skips the baseline comparison for such a run and compare-baseline.py refuses to adopt it.
APPROVAL_DELAY_MS, HUMAN_DELAY_MS	The wait before k6 sends the approval (order-saga) or the human reply (agent-loop).
POLICY_SET=large, POLICY_RUNS	Runs each selected scenario (one, several, or all) with authorization policy enforcement on, as POLICY_RUNS pairs (default 3, a positive whole number; keep it odd). Each pair runs <scenario>+enforce.N with no policy set, and <scenario>+policy.N with a 500-rule Rego policy set, each on a fresh stack. The pair order swaps every round. The run fails when the median of the pair ratios is more than 1.25. A ratio is the p95 of the first SLO metric with the set, divided by the p95 without it.
SWEEP=20,50,100	Runs each named scenario at each rate, from low to high, each on a fresh stack. It stops after the first rate that fails. It writes sweep.json with the highest rate that passed. A scenario with a fixed ramp (webhook-burst) is rejected.
SOAK=1	order-saga only. One long run (DURATION defaults to 60m) at half the SLO rate. It samples the server goroutines and heap. It fails when they grow, or when the late latency is worse than the early latency.

SCENARIO=all does not include SWEEP or SOAK. They run only for a named scenario. Runs with POLICY_SET, SWEEP, or SOAK use their own labels, such as order-saga+policy.1 or order-saga@50, so they never replace the results of a plain run.

Reading k6 Output

http_req_duration...: avg=<n>ms  min=<n>µs  med=<n>ms  max=<n>ms  p(90)=<n>ms  p(95)=<n>ms
http_req_failed.....: <n>%  <n> out of <n>
http_reqs...........: <n>  <n>/s

k6 JSON results are saved below tests/loadtest/results/ in a unique run directory. Each scenario directory contains heap-before.prof, heap-after.prof, goroutine-before.prof, and goroutine-after.prof. Each run directory also has a summary.md with one table for each scenario. One run directory holds one backend.

Recording a Run

After a make loadtest run on both backends, record it:

make loadtest-record

The command takes the newest run directory and the run on the other backend from the same commit and command. It adds one entry to the results log, a file in the repository, and updates its table of the latest result for each scenario and backend. The entry shows the two backends side by side. It has the date, the commit, the machine, the command, and the SLO result for each scenario and backend.

Baseline Comparison

After a full-suite run, run.sh compares the results against the committed baseline for the backend it ran on, tests/loadtest/baseline-sqlite.json or tests/loadtest/baseline-postgres.json, and prints the deltas. Each run directory records its backend in a backend file, and the comparator takes the backend from there, so a PostgreSQL run can never be compared against or written over the SQLite baseline. Regressions above 30% are advisory in the runner; the SLO limits and the audit determine success. A partial run (./run.sh order-saga) skips the comparison, and so does a run with POLICY_SET.

The comparator reads the scenario list from slo.json. For each scenario it compares the p95 of the first SLO metric and the p95 of http_req_duration. It requires all of these results. A missing, zero, negative, or nonfinite latency is rejected before a baseline can be replaced. Adopt a new baseline from a complete run with:

tests/loadtest/compare-baseline.py tests/loadtest/results/{timestamp} --update

Adopt a baseline for each backend. The baselines are committed so changes can be reviewed. Metrics absent from a baseline are reported as new until a complete measured run supplies them.

Profiling After Load Tests

# Compare heap before/after load
go tool pprof -diff_base results/{timestamp}/{scenario}/heap-before.prof \
  results/{timestamp}/{scenario}/heap-after.prof

# Check goroutine state
go tool pprof results/{timestamp}/{scenario}/goroutine-after.prof

Grafana Dashboard

The Grafana performance dashboard is deploy/docker-compose/cluster/grafana/dashboards/ironflow-performance.json. The Docker Compose monitoring profiles provision it. For your own Grafana, import the JSON file. Requires IRONFLOW_METRICS_ENABLED=true.