Ironflow DB and NATS Ownership

Ironflow DB and NATS Ownership An architecture diagram generated by Archify. events · kept forever · PostgreSQL / SQLite — source of truth events kept forever runs + steps · kept forever · PostgreSQL / SQLite — source of truth runs + steps kept forever entity streams · kept forever · PostgreSQL / SQLite — source of truth entity streams kept forever dispatch_queue · the only dispatch path · PostgreSQL / SQLite — source of truth · replaced STEPS (ADR 0037) dispatch_queue the only dispatch path replaced STEPS (ADR 0037) EVENTS · Limits, 7d, 512MB · NATS JetStream — transport, every message expires · kept after ack EVENTS Limits, 7d, 512MB kept after ack STEPS · WorkQueue, 1d, 512MB · NATS JetStream — transport, every message expires · no traffic today STEPS WorkQueue, 1d, 512MB no traffic today RESULTS · Limits, 1h, 128MB · NATS JetStream — transport, every message expires RESULTS Limits, 1h, 128MB PUBSUB · Limits, 7d, 512MB · NATS JetStream — transport, every message expires · public.> PUBSUB Limits, 7d, 512MB public.> CANCEL · Limits, 1h, 64MB · NATS JetStream — transport, every message expires CANCEL Limits, 1h, 64MB SYS_* KV buckets · no DB counterpart · NATS KV — native, no DB table behind it · cron, config, secrets SYS_* KV buckets no DB counterpart cron, config, secrets APP_* KV buckets · user key/value · NATS KV — native, no DB table behind it APP_* KV buckets user key/value ingest inserts event run finished notify subscribers PostgreSQL / SQLite — source of truth NATS JetStream — transport, every message expires NATS KV — native, no DB table behind it Legend Durable record NATS KV bucket JetStream stream

When the database owns the record

  • • emit(), append() and run results write to the DB before publishing
  • • External ingest arrives in NATS first; EventRouter inserts the event and creates runs
  • • An ingest message lost before consumption has no database copy

Message fate

  • • EVENTS, PUBSUB: age > 7d or size > 512MB — NOT deleted on ack
  • • RESULTS: age > 1h / 128MB. CANCEL: age > 1h / 64MB
  • • STEPS is WorkQueue — a message would be deleted on ack, but nothing publishes there

STEPS is created, not used

  • • Since ADR 0037 (#1206) capacity admission is the only dispatch path
  • • Runs enter dispatch_queue and workers long-poll GET /api/v1/workers/{id}/jobs
  • • Nothing publishes to or consumes from ironflow.*.*.steps.> today