NATS JetStream Streams

Ironflow uses 5 JetStream streams to separate concerns: event ingestion, step dispatch, result delivery, real-time notifications, and cancellation signals. Each stream has distinct retention semantics tuned to its role.

┌─────────────────────────────────────────────────────────────────────┐
│  5 NATS JETSTREAM STREAMS                                           │
│                                                                     │
│  EVENTS   LimitsPolicy  7d   ironflow.*.*.events.>      512MB prod  │
│  STEPS    WorkQueue     1d   ironflow.*.*.steps.>        512MB prod │
│  RESULTS  LimitsPolicy  1h   ironflow.*.*.results.>      128MB prod │
│  PUBSUB   LimitsPolicy  7d   public.>                    512MB prod │
│  CANCEL   LimitsPolicy  1h   ironflow.*.*.cancel.>        64MB prod │
└─────────────────────────────────────────────────────────────────────┘

Subject wildcards *.* match {projectName}.{envName} segments.

NATS KV Buckets

KV buckets are separate from streams — purely NATS-native (JetStream KV API), no corresponding DB tables.

┌─────────────────────────────────────────────────────────────────┐
│  System KV buckets (SYS_* prefix, hidden from users)            │
│                                                                 │
│  SYS_circuit_breakers  — CB state per functionID|endpointURL    │
│  SYS_cron_triggers     — cron dedup (first kv.Create() wins)    │
│  SYS_config_*          — env-scoped JSON config docs            │
│  SYS_secrets_*         — encrypted env-scoped secrets           │
│  SYS_debounce_state    — pending debounce arms (per fn+key)     │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  User KV buckets (APP_* prefix, visible in dashboard)           │
│                                                                 │
│  APP_{bucketName}  — arbitrary user key/value                   │
│  Each bucket = 1 NATS KV bucket (backed by a JetStream stream)  │
└─────────────────────────────────────────────────────────────────┘

DB vs NATS: Ownership Boundaries

DB is the source of truth and audit trail. NATS is transport, work queue, and ephemeral real-time layer.

DB (Postgres / SQLite)          NATS JetStream
────────────────────────        ───────────────────────────────────
events          (forever)  ←→  EVENTS stream (7d, then gone)
runs            (forever)  ←   STEPS stream (deleted on ACK)
steps           (forever)  ←   RESULTS stream (1h, then gone)
functions       (forever)      PUBSUB stream (7d, notifications)
entity streams  (forever)      CANCEL stream (1h, control only)
consumer_groups (forever)      KV buckets (no DB counterpart)
audit_events    (forever)

Message Fate

Stream    Deleted when?
────────  ─────────────────────────────────────────────────────
EVENTS    age > 7d or size > 512MB  (NOT on ACK — stays after read)
STEPS     immediately on ACK        (WorkQueue — consumed exactly once)
RESULTS   age > 1h or size > 128MB
PUBSUB    age > 7d or size > 512MB
CANCEL    age > 1h or size > 64MB