Skip to content
ironflow
Ironflow is pre-1.0: APIs can change between releases. Pin a version. What is ready →

Projections

Projections build read-optimized materialized views from event streams. Instead of reading an entity’s full event history every time, a projection continuously processes events and maintains a derived state you can query instantly.

How It Works

Section titled “How It Works”

Projections are a split responsibility between your SDK worker and the Ironflow server:

  • Your app (SDK) — defines the projection handler and runs it in a streaming or poll loop
  • Ironflow server — stores state, manages NATS consumers, serves queries to your frontend

The SDK registers the projection, receives events (via real-time streaming or polling), runs your handler locally, and saves the resulting state back to the server. The server never executes your handler logic — it just feeds events and persists whatever state the SDK sends back.

This means your projection logic lives in your codebase, versioned with your app, while the server handles durability, partitioning, lag tracking, and real-time distribution to connected clients.

Projection Derivation — showing events flowing through a reducer to build read-optimized state

Entity identity in handler events

Section titled “Entity identity in handler events”

Entity stream events expose entityId, entityType, and entityVersion in JavaScript; EntityID, EntityType, and EntityVersion in Go; and entity_id, entity_type, and entity_version in Python. These are the canonical stream identity and per-entity version, carried separately from event data and user metadata. They are the same during live delivery and history rebuilds, using polling or streaming. Ordinary events leave them undefined in JavaScript, None in Python, and zero-valued in Go.

seq (Seq in Go) remains the global projection delivery cursor. It is not the entity version. Use the entity fields to identify a seat or other domain entity without copying server-owned identity into the event payload.

Two Modes

Section titled “Two Modes”
ModeState locationHandler typeUse case
ManagedIronflow DBPure reducer (state, event) => newStateDashboards, aggregations, read models
ExternalYour databaseSide-effect handler (event) => voidSearch indexes, cache warming, third-party sync

Mode is auto-detected: if you provide initialState, it’s managed. Otherwise it’s external. You can also set mode explicitly.

Managed Projections

Section titled “Managed Projections”

A managed projection is a pure reducer — it takes the current state and an event, and returns the new state. Ironflow stores the materialized state in its database.

import { createProjection } from "@ironflow/node";


const orderStats = createProjection({
  name: "order-stats",
  events: ["order.created", "order.completed"],
  handler: (state, event) => {
    switch (event.name) {
      case "order.created":
        return {
          ...state,
          totalOrders: state.totalOrders + 1,
          totalAmount: state.totalAmount + event.data.amount,
        };
      case "order.completed":
        return { ...state, completedOrders: state.completedOrders + 1 };
      default:
        return state;
    }
  },
  initialState: () => ({
    totalOrders: 0,
    completedOrders: 0,
    totalAmount: 0,
  }),
});

Querying State

Section titled “Querying State”

Once a managed projection is running, query its state via the REST API or SDK:

import { ironflow } from "@ironflow/browser";


const result = await ironflow.getProjection<OrderStats>("order-stats");
console.log(result.state); // { totalOrders: 42, completedOrders: 38, totalAmount: 12500 }
console.log(result.mode); // "managed"

External Projections

Section titled “External Projections”

An external projection runs side effects — writing to your own database, updating a search index, or syncing to a third-party service. Ironflow only tracks the cursor position (which events have been processed).

import { createProjection } from "@ironflow/node";


const searchIndex = createProjection({
  name: "employee-search-index",
  events: ["employee.*"],
  handler: async (event, ctx) => {
    // Side effect — write to your own system
    await elasticsearch.index({
      index: "employees",
      id: event.data.id,
      body: event.data,
    });
    ctx.logger.info(`Indexed employee ${event.data.id}`);
  },
  // No initialState → auto-detected as external mode
});

Registering Projections

Section titled “Registering Projections”

Add projections to your worker configuration. Projections need a persistent poll loop, so they run in pull mode (workers), not push mode (serverless).

import { createWorker } from "@ironflow/node";


const worker = createWorker({
  functions: [processOrder, sendEmail],
  projections: [orderStats, searchIndex],
});


// Projections register and start polling as the worker starts. start() blocks
// until the worker is stopped — it does not resolve once polling is under way,
// so put any setup code before it, not after.
await worker.start();

Run as many worker replicas as you like. Each projection has one exclusive runner lease (ADR 0120): the replica that holds it reads events, and the others wait as hot standbys and take over if it stops or loses the lease.

Streaming vs Polling

Section titled “Streaming vs Polling”

The projection runner supports two event delivery strategies:

StrategyHow it worksLatencyAvailability
StreamingOpens a persistent ConnectRPC server-stream; forwards the first event and any available backlog without waiting to fill a batchNo batch-fill delay; depends on server load and networkIronflow ≥ 0.15
PollingPeriodically calls PollProjectionEvents with exponential backoff (1 s → 2 s → 4 s → 8 s → 10 s, reset on success)SecondsAll versions

The Go, TypeScript and Python workers automatically try streaming first. If the server does not support the stream (older version), they fall back to polling transparently — no code changes required.

An idle server pull waits up to 1 second for the first event, returning as soon as it arrives. This timeout does not delay an event that is already available. In local embedded-NATS regression tests, a single event reached the streaming client in under 10 ms; this is a measurement, not a latency guarantee. Handler execution and the SDK micro-batch state-save delay below add to the time before a projection read reflects the event.

Micro-batching (streaming mode)

Section titled “Micro-batching (streaming mode)”

In streaming mode the runner groups incoming events into micro-batches to amortize state saves:

  • Batch size trigger — flushes when the batch reaches batchSize events (default 100)
  • Time trigger — flushes 100 ms after the first event in the batch, whichever comes first

This means a burst of 50 events results in a single state save instead of 50 individual writes.

Reconnection

Section titled “Reconnection”

If the stream ends cleanly (e.g., server restart), the runner reconnects after a 1-second delay. On stream errors it backs off exponentially from 2 s, capped at 30 s, and resets on the next successful read — a flat retry against an engine that is down produces an error line per projection per attempt, forever. Either way it resumes from its delivery position, which rises only after a batch is saved or acknowledged, so a batch that failed comes again and a saved one does not (ADR 0164). No events are lost.

Partitioned Projections

Section titled “Partitioned Projections”

Partition projections by a key extracted from event data. Each partition maintains its own independent state — useful for per-customer, per-tenant, or per-entity projections.

const customerStats = createProjection({
  name: "customer-stats",
  events: ["order.created", "order.completed"],
  partitionKey: "$.customerId",
  handler: (state, event) => ({
    ...state,
    orderCount: state.orderCount + 1,
    totalSpend: state.totalSpend + (event.data.amount ?? 0),
  }),
  initialState: () => ({ orderCount: 0, totalSpend: 0 }),
});

Query a specific partition:

const result = await ironflow.getProjection("customer-stats", {
  partition: "cust-456",
});

The partitionKey uses dot-notation JSONPath to extract the key from event data. The path is rooted at the event payload you emitted, so $.customerId extracts cust-456 from { "customerId": "cust-456" }. An event whose path does not match goes to the __global__ partition.

Real-Time Subscriptions

Section titled “Real-Time Subscriptions”

Subscribe to projection state updates in the browser. Updates are pushed over the subscription connection whenever the projection state changes.

import { ironflow } from "@ironflow/browser";


// Subscribe to all updates for a projection
const sub = await ironflow.subscribeToProjection<OrderStats>("order-stats", {
  onUpdate: (state, event) => {
    console.log("Projection updated:", state);
  },
  onError: (error) => console.error(error),
});


// Subscribe to a specific partition
const partSub = await ironflow.subscribeToProjection(
  "customer-stats",
  {
    onUpdate: (state) => console.log("Customer updated:", state),
  },
  { partition: "cust-456" },
);


// Clean up
sub.unsubscribe();
partSub.unsubscribe();

Rebuilding Projections

Section titled “Rebuilding Projections”

Projection consumers retry failed delivery without a delivery-count limit. Metadata lookup failures return the fetched batch for retry and do not advance the projection cursor. Permanently malformed envelopes are terminated and skipped within the same pull, so they cannot hide a valid event behind them. A check bounded to 64 global PUBSUB sequence positions, including other subjects and environments, avoids recreating the consumer for a malformed-only tail after the saved cursor. Wider gaps, readable events, and uncertain lookups retain conservative replay so unsaved valid events remain recoverable. Consumers created by older builds are updated in place when a worker registers or re-registers: restart or re-register projection workers after upgrading the engine. Messages already exhausted under an old delivery limit need a rebuild to become available again.

Rebuild a projection to reprocess all events from the beginning. This is useful when you change your handler logic or need to recover from errors.

await client.projections.rebuild("order-stats");

Only a projection in active or error status can be rebuilt, and one whose PUBSUB stream was recreated cannot be: delete it and register or create it again. After a failed rebuild of a handler-based projection, every read answers failed_precondition and the runner stops, until a rebuild succeeds or the projection is deleted. A client’s read of its state and a catch-up wait answer the same until a new rebuild starts. Fix the cause, start the rebuild without --to or --partition, then restart the runner. After a failed or cancelled rebuild of a SQL projection, no live event applies and every query answers failed_precondition, until a new rebuild starts or the projection is deleted and created again. What happens during a rebuild:

  1. Projection status changes to rebuilding
  2. All stored state is deleted (managed mode)
  3. The rebuild manager resolves a start cursor (0 for full rebuild, or the nats_seq of fromEventId for partial) and records progress markers on the projection registry
  4. The SDK continues its normal stream/poll loop; the server replays historical events from the cursor with phase=scan (on spans projection.rebuild.batch and the ironflow_projection_rebuild_events_applied_total counter) while live events keep flowing
  5. Status returns to active once the replay is drained and the runner has saved or acknowledged every replayed event. The cursor stays at the last event the runner committed; the rebuild does not move it to the target

The runner’s delivery position keeps the replay and the live events from overlapping, so no event is applied twice (ADR 0164). A cancelled rebuild hands the rest of its range to the live stream, which still holds only the events inside its retention; rebuild again if the cancelled range reaches past it.

Pause and Resume

Section titled “Pause and Resume”

Temporarily pause a projection to stop it from processing new events, then resume when ready.

if err := client.Projections().Pause(ctx, "order-stats"); err != nil { log.Fatal(err) }
if err := client.Projections().Resume(ctx, "order-stats"); err != nil { log.Fatal(err) }

A projection can only be paused when it is active or rebuilding. It can only be resumed when paused.

Deleting Projections

Section titled “Deleting Projections”

Unregister a projection and remove its stored state:

if err := client.Projections().Delete(ctx, "order-stats"); err != nil { log.Fatal(err) }

Projection Status

Section titled “Projection Status”

Check the status of a projection:

const status = await ironflow.getProjectionStatus("order-stats");
console.log(status.status); // "active" | "rebuilding" | "paused" | "error"
console.log(status.lastEventSeq); // last processed event sequence
console.log(status.errorMessage); // error details if status is "error"

Status Values

Section titled “Status Values”
StatusDescription
activeProcessing events normally
rebuildingReprocessing from the beginning
pausedTemporarily stopped
errorFailed — check errorMessage for details

Listing Projections

Section titled “Listing Projections”

List all registered projections:

const projections = await ironflow.listProjections();
projections.forEach((p) => {
  console.log(`${p.name}: ${p.status} (${p.mode})`);
});

SQL Projections

Section titled “SQL Projections”

SQL projections are materialized, like handler-based projections. The state lives in a real SQL table inside Ironflow’s own database. A parameterized INSERT/UPDATE/DELETE writes it for each event, instead of a reducer function.

The engine runs them. Every node keeps a consumer for each SQL projection, NATS delivers each event to one node, and that node applies the handler SQL and moves the cursor in one transaction. No worker and no poll is needed. A row appears soon after the event reaches the stream. The engine applies events in stream order, which is not always the order in which you emitted them. Plain Emit events have no fixed order, so do not write handler SQL that needs one event before another. Let each event upsert only its own columns, for example paid_at for order.paid, instead of an UPDATE that needs the row to exist.

The table you declare is always proj_<projection name>. Ironflow keeps the rows in an environment-scoped table behind that name, so two environments can each own a projection with the same name. Applications never connect to that table directly; it is reached only through the API.

Create one with the ConnectRPC CreateSQLProjection RPC (there is no REST route for this):

await client.sqlProjections.create({
  name: "active_orders",
  tableSql: `
    CREATE TABLE proj_active_orders (
      order_id TEXT PRIMARY KEY,
      customer_id TEXT NOT NULL,
      total NUMERIC(12,2)
    );
    CREATE INDEX proj_active_orders_customer ON proj_active_orders (customer_id);
  `,
  events: ["order.placed", "order.cancelled"],
  eventHandlers: {
    "order.placed":
      "INSERT INTO proj_active_orders (order_id, customer_id, total) VALUES (:entity_id, :data.customerId, :data.total)",
    "order.cancelled":
      "DELETE FROM proj_active_orders WHERE order_id = :entity_id",
  },
});
from ironflow.rpc.v1 import CreateSQLProjectionRequest


await rpc.projections.create_sql(CreateSQLProjectionRequest(
    name="active_orders",
    table_sql="""
        CREATE TABLE proj_active_orders (
          order_id TEXT PRIMARY KEY,
          customer_id TEXT NOT NULL,
          total NUMERIC(12,2)
        );
        CREATE INDEX proj_active_orders_customer ON proj_active_orders (customer_id);
    """,
    events=["order.placed", "order.cancelled"],
    event_handlers={
        "order.placed":
            "INSERT INTO proj_active_orders (order_id, customer_id, total) VALUES (:entity_id, :data.customerId, :data.total)",
        "order.cancelled":
            "DELETE FROM proj_active_orders WHERE order_id = :entity_id",
    },
))

Read it back with QuerySQLProjection — not POST /ironflow.v1.ProjectionService/GetProjection, which returns the projection’s registry entry and cursor, not its rows:

const result = await client.sqlProjections.query("active_orders", {
  where: "customer_id = 'cust-1'",
  orderBy: "total DESC",
  limit: 50,
});
from ironflow.rpc.v1 import QuerySQLProjectionRequest


result = await rpc.projections.query_sql(QuerySQLProjectionRequest(
    name="active_orders",
    where="customer_id = 'cust-1'",
    order_by="total DESC",
    limit=50,
))

SQL projections:

  • Have type "sql" in the projection listing (vs "sdk" for handler-based projections)
  • Are driven by the engine. PollProjectionEvents on a SQL projection returns the cursor and processes nothing; StreamProjectionEvents refuses it
  • Rebuild from the start only: --partition, --from and --to are refused (partition, from_event_id and to_event_id in the RPC), because the table cannot be cleared in part and the live consumer has already acked the later events. A failed or cancelled rebuild leaves status error, because the read model is partial: no live event applies and queries answer failed_precondition until a new rebuild starts. Any node can cancel it, also one that did not start it
  • Are bound to the PUBSUB stream they were created against. If that stream is recreated, such as a wiped NATS data directory beside a kept database, the projection stops with status error within 30 seconds, and a rebuild is refused. Run ironflow projection delete <name>, then ironflow projection create with the same definition; the new read model holds the current stream’s retained events. A SQL projection created before this binding existed stops the same way on upgrade
  • Need NATS 2.14.7 or later on every server in cluster mode
  • Own exactly one proj_-prefixed table, named after the projection. The name must therefore be an unquoted SQL identifier — lowercase letters, digits and underscores, not starting with a digit, at most 45 characters — so active_orders, not active-orders or Active_Orders. Handler-based ("sdk") projections have no such restriction. A name that starts with ironflow_ is reserved for both kinds. The DDL may also declare indexes on that table, and CREATE EXTENSION IF NOT EXISTS vector / pg_trgm before it (PostgreSQL only)
  • Cannot read or write any other table — the DDL, the event handlers, and the query clauses are each parsed against a closed grammar, and every value is bound rather than concatenated. In the DDL and in a handler that means a ( may only follow the projection’s own table, a SQL keyword that takes one, a type modifier, or a function on a closed list, so a DEFAULT or a VALUES entry cannot call something like table_to_xml('other_table'::regclass, …) (#1689)
  • Return rows twice: rows with every value stringified, and typedRows with the SQL type preserved
  • Accept a fixed set of expressions in where and orderBy beyond ordinary filters — @@ with plainto_tsquery, pgvector distance (<->, <=>, <#>), and ts_rank(...) — so a vector or full-text read model can reach its index. Those need the PostgreSQL backend

See the browser SDK reference for the full where and orderBy grammar.

Configuration Options

Section titled “Configuration Options”
OptionDefaultDescription
name(required)Unique projection name: letters, digits, _ and -, with no -- and no ironflow_ prefix
events(required)Event names to subscribe to. Both * and > match the remainder of the subject (order.* and order.> are equivalent — Ironflow rewrites * to > before creating the NATS consumer).
handler(required)Handler function (reducer for managed, side-effect for external)
initialState—Factory function returning initial state (triggers managed mode)
modeauto-detected"managed" or "external"
partitionKey—JSONPath for partition key extraction (e.g., "$.customerId")
maxRetries3Accepted and defaulted, but not used by any SDK runner and not sent to the server — see Error Handling
batchSize100Number of events to fetch per poll

HTTP API Reference

Section titled “HTTP API Reference”
MethodEndpointDescription
POST/ironflow.v1.ProjectionService/ListProjectionsList all projections
POST/ironflow.v1.ProjectionService/GetProjectionGet projection state
POST/ironflow.v1.ProjectionService/GetProjectionStatusGet projection status
GET/api/v1/projections/{name}/partitionsList partitions (partitioned mode)
POST/ironflow.v1.ProjectionService/WaitProjectionCatchupLong-poll until the projection reaches a minimum NATS sequence (minSeq)
POST/ironflow.v1.ProjectionService/WaitProjectionCatchupBatchWait for several projections to catch up in one call
POST/ironflow.v1.ProjectionService/WaitProjectionCatchupStreamServer-stream variant of WaitProjectionCatchup — progress frames instead of one blocking response
POST/ironflow.v1.ProjectionService/CreateSQLProjectionCreate a SQL projection (see SQL Projections)
POST/ironflow.v1.ProjectionService/QuerySQLProjectionQuery a SQL projection’s rows
POST/ironflow.v1.ProjectionService/WaitForEventBlock until a specific event has been applied by a projection
POST/ironflow.v1.ProjectionService/RebuildProjectionTrigger rebuild
POST/ironflow.v1.ProjectionService/GetRebuildJobGet rebuild job progress
POST/ironflow.v1.ProjectionService/CancelRebuildCancel an in-progress rebuild
POST/ironflow.v1.ProjectionService/ListProjectionHistoryList archived definitions captured before a replacement or deletion
POST/ironflow.v1.ProjectionService/GetProjectionRevisionGet one archived definition by revision
POST/ironflow.v1.ProjectionService/PauseProjectionPause projection
POST/ironflow.v1.ProjectionService/ResumeProjectionResume projection
DELETE/api/v1/projections/{name}Delete projection

The SDKs wrap the two history RPCs: Projections().ListHistory and GetRevision in Go, client.projections.listHistory and getRevision in Node, ironflow.listProjectionHistory and getProjectionRevision in the browser, and rpc.projections.list_history and get_revision in Python. History is newest first. The current definition is not in it. limit defaults to 50 with a maximum of 200. Pass nextBeforeRevision from one page as beforeRevision for the next page.

The table omits the worker-facing RPCs that the SDK runners call for you: RegisterProjection, UnregisterProjection, AcquireProjectionLease, RenewProjectionLease, ReleaseProjectionLease, PollProjectionEvents, StreamProjectionEvents, AckProjectionEvents and SaveProjectionState.

JSON request fields for POST /ironflow.v1.ProjectionService/GetProjection (include name):

  • partition — Return state for a specific partition (default: __global__)

Error Handling

Section titled “Error Handling”

Handler-based ("sdk") and SQL projections fail in opposite directions — retry forever versus skip and record.

Handler-based projections. A throwing handler aborts the batch before the state save (managed) or the ack (external), so nothing advances and the same events are redelivered on the next poll. The runner logs the error and backs off (1 s → 10 s), then tries again — indefinitely. A poison event stalls the projection until you fix and redeploy the handler, or rebuild past it.

Two things the runner does not currently do, despite what the surrounding API suggests:

  • maxRetries (max_retries in Python) is accepted and defaulted to 3, but no runner reads it, no SDK sends it at registration (the registration message has no such field), and no server path uses it. There is no per-event retry counter.
  • No runner flips the projection to error or writes errorMessage on a handler failure, and none treats NonRetryableError differently from any other throw.

SQL projections. These execute server-side, and a failing statement is handled the other way round: the event is logged, acked, and skipped so the cursor still advances — one poison event cannot stall the projection. The registry is flipped to error with errorMessage set to the first failure (event "<name>" (seq N): <err>), visible through ListProjections, POST /ironflow.v1.ProjectionService/GetProjectionStatus, and the dashboard. Later events keep processing. An infrastructure error (database or NATS) is retried and never skips an event. A message that cannot be decoded is recorded like a handler failure. Decode and handler failures are also retained in dead_letters atomically with cursor advancement, without application bytes. A message with unreadable NATS metadata stays retryable and is counted in the parse metric.

Managed vs External — When to Use Which

Section titled “Managed vs External — When to Use Which”
ScenarioModeWhy
Dashboard aggregationsManagedQuery state directly from Ironflow
Search indexExternalData lives in Elasticsearch/Algolia
Cache warmingExternalData lives in Redis/Memcached
Reporting summariesManagedSimple reducer, queryable via API
Third-party syncExternalSide effects to external systems
Per-customer analyticsManaged + partitionedPartitioned state in Ironflow