Skip to content
ironflow

Projections

Projections build read-optimized materialized views from event streams. Instead of reading an entity’s full event history every time, a projection continuously processes events and maintains a derived state you can query instantly.

How It Works

Projections are a split responsibility between your SDK worker and the Ironflow server:

  • Your app (SDK) — defines the projection handler and runs it in a streaming or poll loop
  • Ironflow server — stores state, manages NATS consumers, serves queries to your frontend

The SDK registers the projection, receives events (via real-time streaming or polling), runs your handler locally, and saves the resulting state back to the server. The server never executes your handler logic — it just feeds events and persists whatever state the SDK sends back.

This means your projection logic lives in your codebase, versioned with your app, while the server handles durability, partitioning, lag tracking, and real-time distribution to connected clients.

Projection Derivation — showing events flowing through a reducer to build read-optimized state

Two Modes

ModeState locationHandler typeUse case
ManagedIronflow DBPure reducer (state, event) => newStateDashboards, aggregations, read models
ExternalYour databaseSide-effect handler (event) => voidSearch indexes, cache warming, third-party sync

Mode is auto-detected: if you provide initialState, it’s managed. Otherwise it’s external. You can also set mode explicitly.

Managed Projections

A managed projection is a pure reducer — it takes the current state and an event, and returns the new state. Ironflow stores the materialized state in its database.

import { createProjection } from "@ironflow/node";


const orderStats = createProjection({
  name: "order-stats",
  events: ["order.created", "order.completed"],
  handler: (state, event) => {
    switch (event.name) {
      case "order.created":
        return {
          ...state,
          totalOrders: state.totalOrders + 1,
          totalAmount: state.totalAmount + event.data.amount,
        };
      case "order.completed":
        return { ...state, completedOrders: state.completedOrders + 1 };
      default:
        return state;
    }
  },
  initialState: () => ({
    totalOrders: 0,
    completedOrders: 0,
    totalAmount: 0,
  }),
});

Querying State

Once a managed projection is running, query its state via the REST API or SDK:

import { ironflow } from "@ironflow/browser";


const result = await ironflow.getProjection<OrderStats>("order-stats");
console.log(result.state); // { totalOrders: 42, completedOrders: 38, totalAmount: 12500 }
console.log(result.mode); // "managed"

Defining a projection is Go/TypeScript only

The Python SDK can read, rebuild, pause, resume, and delete projections, but it cannot define one — a reducer is worker-runtime code and there is no projections_create method. Write the projection in Go or TypeScript and operate it from Python.

External Projections

An external projection runs side effects — writing to your own database, updating a search index, or syncing to a third-party service. Ironflow only tracks the cursor position (which events have been processed).

import { createProjection } from "@ironflow/node";


const searchIndex = createProjection({
  name: "employee-search-index",
  events: ["employee.*"],
  handler: async (event, ctx) => {
    // Side effect — write to your own system
    await elasticsearch.index({
      index: "employees",
      id: event.data.id,
      body: event.data,
    });
    ctx.logger.info(`Indexed employee ${event.data.id}`);
  },
  // No initialState → auto-detected as external mode
});

Registering Projections

Add projections to your worker configuration. Projections need a persistent poll loop, so they run in pull mode (workers), not push mode (serverless).

import { createWorker } from "@ironflow/node";


const worker = createWorker({
  functions: [processOrder, sendEmail],
  projections: [orderStats, searchIndex],
});


// Projections register and start polling as the worker starts. start() blocks
// until the worker is stopped — it does not resolve once polling is under way,
// so put any setup code before it, not after.
await worker.start();

If you pass projections to serve() (push mode), a warning is logged. Projections require a persistent process — use createWorker() instead.

Streaming vs Polling

The projection runner supports two event delivery strategies:

StrategyHow it worksLatencyAvailability
StreamingOpens a persistent ConnectRPC server-stream that pushes events in real-timeMillisecondsIronflow ≥ 0.15
PollingPeriodically calls PollProjectionEvents with exponential backoff (1 s → 2 s → 4 s → 8 s → 10 s, reset on success)SecondsAll versions

The worker automatically tries streaming first. If the server returns 404 or 501 (older version), it falls back to polling transparently — no code changes required.

Micro-batching (streaming mode)

In streaming mode the runner groups incoming events into micro-batches to amortize state saves:

  • Batch size trigger — flushes when the batch reaches batchSize events (default 100)
  • Time trigger — flushes 100 ms after the first event in the batch, whichever comes first

This means a burst of 50 events results in a single state save instead of 50 individual writes.

Reconnection

If the stream ends cleanly (e.g., server restart), the runner reconnects after a 1-second delay. On stream errors the delay is 2 seconds. Either way it resumes from the last acknowledged position. No events are lost.

Partitioned Projections

Partition projections by a key extracted from event data. Each partition maintains its own independent state — useful for per-customer, per-tenant, or per-entity projections.

const customerStats = createProjection({
  name: "customer-stats",
  events: ["order.created", "order.completed"],
  partitionKey: "$.data.customerId",
  handler: (state, event) => ({
    ...state,
    orderCount: state.orderCount + 1,
    totalSpend: state.totalSpend + (event.data.amount ?? 0),
  }),
  initialState: () => ({ orderCount: 0, totalSpend: 0 }),
});

Query a specific partition:

const result = await ironflow.getProjection("customer-stats", {
  partition: "cust-456",
});

The partitionKey uses dot-notation JSONPath to extract the key from event data. For example, $.data.customerId extracts customerId from { "data": { "customerId": "cust-456" } }.

Real-Time Subscriptions

Subscribe to projection state updates in the browser. Events are pushed via WebSocket whenever the projection state changes.

import { ironflow } from "@ironflow/browser";


// Subscribe to all updates for a projection
const sub = await ironflow.subscribeToProjection<OrderStats>("order-stats", {
  onUpdate: (state, event) => {
    console.log("Projection updated:", state);
  },
  onError: (error) => console.error(error),
});


// Subscribe to a specific partition
const partSub = await ironflow.subscribeToProjection(
  "customer-stats",
  {
    onUpdate: (state) => console.log("Customer updated:", state),
  },
  { partition: "cust-456" },
);


// Clean up
sub.unsubscribe();
partSub.unsubscribe();

Rebuilding Projections

Rebuild a projection to reprocess all events from the beginning. This is useful when you change your handler logic or need to recover from errors.

await client.projections.rebuild("order-stats");

What happens during a rebuild:

  1. Projection status changes to rebuilding
  2. All stored state is deleted (managed mode)
  3. The rebuild manager resolves a start cursor (0 for full rebuild, or the nats_seq of fromEventId for partial) and records progress markers on the projection registry
  4. The SDK continues its normal stream/poll loop; the server replays historical events from the cursor with phase=scan (on spans projection.rebuild.batch and the ironflow_projection_rebuild_events_applied_total counter) while live events keep flowing
  5. Status returns to active once the rebuild catches up to the live cursor

For external projections, you’re responsible for clearing your own data store before triggering a rebuild.

Pause and Resume

Temporarily pause a projection to stop it from processing new events, then resume when ready.

if err := client.Projections().Pause(ctx, "order-stats"); err != nil { log.Fatal(err) }
if err := client.Projections().Resume(ctx, "order-stats"); err != nil { log.Fatal(err) }

A projection can only be paused when it is active or rebuilding. It can only be resumed when paused.

Deleting Projections

Unregister a projection and remove its stored state:

if err := client.Projections().Delete(ctx, "order-stats"); err != nil { log.Fatal(err) }

Projection Status

Check the status of a projection:

const status = await ironflow.getProjectionStatus("order-stats");
console.log(status.status); // "active" | "rebuilding" | "paused" | "error"
console.log(status.lastEventSeq); // last processed event sequence
console.log(status.errorMessage); // error details if status is "error"

Status Values

StatusDescription
activeProcessing events normally
rebuildingReprocessing from the beginning
pausedTemporarily stopped
errorFailed — check errorMessage for details

Listing Projections

List all registered projections:

const projections = await ironflow.listProjections();
projections.forEach((p) => {
  console.log(`${p.name}: ${p.status} (${p.mode})`);
});

SQL Projections

SQL projections are materialized, like handler-based projections — they process events. The difference is where the state lives and what writes it: a real SQL table inside Ironflow’s own database, written by a parameterized INSERT/UPDATE/DELETE per event instead of by a reducer function.

The table name is always proj_<projection name>. Applications never connect to that table directly; it is reached only through the API.

Create one with the ConnectRPC CreateSQLProjection RPC (there is no REST route for this):

await client.sqlProjections.create({
  name: "active_orders",
  tableSql: `
    CREATE TABLE proj_active_orders (
      order_id TEXT PRIMARY KEY,
      customer_id TEXT NOT NULL,
      total NUMERIC(12,2)
    );
    CREATE INDEX proj_active_orders_customer ON proj_active_orders (customer_id);
  `,
  events: ["order.placed", "order.cancelled"],
  eventHandlers: {
    "order.placed":
      "INSERT INTO proj_active_orders (order_id, customer_id, total) VALUES (:entity_id, :data.customerId, :data.total)",
    "order.cancelled":
      "DELETE FROM proj_active_orders WHERE order_id = :entity_id",
  },
});

Read it back with QuerySQLProjection — not POST /ironflow.v1.ProjectionService/GetProjection, which returns the projection’s registry entry and cursor, not its rows:

const result = await client.sqlProjections.query("active_orders", {
  where: "customer_id = 'cust-1'",
  orderBy: "total DESC",
  limit: 50,
});

SQL projections:

  • Have type "sql" in the projection listing (vs "sdk" for handler-based projections)
  • Own exactly one proj_-prefixed table, named after the projection. The name must therefore be an unquoted SQL identifier — letters, digits and underscores — so active_orders, not active-orders. Handler-based ("sdk") projections have no such restriction. The DDL may also declare indexes on that table, and CREATE EXTENSION IF NOT EXISTS vector / pg_trgm before it (PostgreSQL only)
  • Cannot read or write any other table — the DDL, the event handlers, and the query clauses are each parsed against a closed grammar, and every value is bound rather than concatenated. In the DDL and in a handler that means a ( may only follow the projection’s own table, a SQL keyword that takes one, a type modifier, or a function on a closed list, so a DEFAULT or a VALUES entry cannot call something like table_to_xml('other_table'::regclass, …) (#1689)
  • Return rows twice: rows with every value stringified, and typedRows with the SQL type preserved
  • Accept a fixed set of expressions in where and orderBy beyond ordinary filters — @@ with plainto_tsquery, pgvector distance (<->, <=>, <#>), and ts_rank(...) — so a vector or full-text read model can reach its index. Those need the PostgreSQL backend

See the browser SDK reference for the full where and orderBy grammar.

Configuration Options

OptionDefaultDescription
name(required)Unique projection name
events(required)Event names to subscribe to. Both * and > match the remainder of the subject (order.* and order.> are equivalent — Ironflow rewrites * to > before creating the NATS consumer).
handler(required)Handler function (reducer for managed, side-effect for external)
initialState—Factory function returning initial state (triggers managed mode)
modeauto-detected"managed" or "external"
partitionKey—JSONPath for partition key extraction (e.g., "$.data.customerId")
maxRetries3Accepted and registered, but not yet enforced by either runner — see Error Handling
batchSize100Number of events to fetch per poll

HTTP API Reference

MethodEndpointDescription
POST/ironflow.v1.ProjectionService/ListProjectionsList all projections
POST/ironflow.v1.ProjectionService/GetProjectionGet projection state
POST/ironflow.v1.ProjectionService/GetProjectionStatusGet projection status
GET/api/v1/projections/{name}/partitionsList partitions (partitioned mode)
POST/ironflow.v1.ProjectionService/WaitProjectionCatchupLong-poll until the projection reaches a minimum NATS sequence (minSeq)
POST/ironflow.v1.ProjectionService/WaitProjectionCatchupBatchWait for several projections to catch up in one call
POST/ironflow.v1.ProjectionService/WaitForEventBlock until a specific event has been applied by a projection
POST/ironflow.v1.ProjectionService/RebuildProjectionTrigger rebuild
POST/ironflow.v1.ProjectionService/GetRebuildJobGet rebuild job progress
POST/ironflow.v1.ProjectionService/CancelRebuildCancel an in-progress rebuild
POST/ironflow.v1.ProjectionService/PauseProjectionPause projection
POST/ironflow.v1.ProjectionService/ResumeProjectionResume projection
DELETE/api/v1/projections/{name}Delete projection

JSON request fields for POST /ironflow.v1.ProjectionService/GetProjection (include name):

  • partition — Return state for a specific partition (default: __global__)

Error Handling

Handler-based ("sdk") and SQL projections fail in opposite directions — retry forever versus skip and record.

Handler-based projections. A throwing handler aborts the batch before the state save (managed) or the ack (external), so nothing advances and the same events are redelivered on the next poll. The runner logs the error and backs off (1 s → 10 s), then tries again — indefinitely. A poison event stalls the projection until you fix and redeploy the handler, or rebuild past it.

Two things the runner does not currently do, despite what the surrounding API suggests:

  • maxRetries is accepted, defaulted to 3, and sent to the server at registration, but no runner and no server path reads it. There is no per-event retry counter.
  • Neither runner flips the projection to error or writes errorMessage on a handler failure, and neither treats NonRetryableError differently from any other throw.

SQL projections. These execute server-side, and a failing statement is handled the other way round: the event is logged, acked, and skipped so the cursor still advances — one poison event cannot stall the projection. The registry is flipped to error with errorMessage set to the first failure (event "<name>" (seq N): <err>), visible through ListProjections, POST /ironflow.v1.ProjectionService/GetProjectionStatus, and the dashboard. Later events keep processing.

Managed vs External — When to Use Which

ScenarioModeWhy
Dashboard aggregationsManagedQuery state directly from Ironflow
Search indexExternalData lives in Elasticsearch/Algolia
Cache warmingExternalData lives in Redis/Memcached
Reporting summariesManagedSimple reducer, queryable via API
Third-party syncExternalSide effects to external systems
Per-customer analyticsManaged + partitionedPartitioned state in Ironflow

Domain-Driven Design

Projections implement the Read Model side of CQRS — separate, query-optimized views derived from domain events. See CQRS with Projections for how this maps to Domain-Driven Design.