Skip to content
Ironflow

Exporting Customer Data

GET /api/v1/export streams a snapshot of your organization as newline-delimited JSON (NDJSON). It serves two needs from the same endpoint: data portability (offboarding, migration) and forensic export (incident response, audit replay).

Quick Start

Terminal window
curl -N -H "Authorization: Bearer $IRONFLOW_API_KEY" \
  https://your-cluster.ironflow.cloud/api/v1/export \
  > export.ndjson

-N disables curl’s output buffering so heartbeat lines arrive in real time on long streams.

Authorization

Admin-only. The endpoint maps to the org:export action, granted exclusively to the admin role. Developer and viewer API keys receive 403 Forbidden.

Platform operators export a tenant org via the standard impersonation header (X-Ironflow-Org); no separate ?org= parameter exists.

What gets exported

v1 includes four entities. Default: all four.

Entity Source
org The caller’s organization row
projects All projects under the org
environments All environments across the org’s projects
audit_events Every audit row in every environment, snapshot-bounded

What is intentionally NOT exported

Excluded Why
Secret values Bounded by the cluster DEK — never leaves the engine
Secret ciphertext Same reasoning; a stolen export must not be offline-crackable
API key hashes apikeys.key_hash is a credential, not a portable record
JWT signing secret Rotates via the cluster-token push path, not the data plane
KV bucket values Customer code may write credentials there; opt-in only
Run / event / entity-stream rows Not walked as separate entities in v1 — tracked as a follow-on

If you need run, event, or entity-stream content portability as their own records, file an issue against #774 follow-ons.

Step output is in audit_event payloads

step.completed audit rows include the step’s output field in the payload. If your step functions return secrets, tokens, or PII as their output, those values land in audit_events records and therefore in the export. This is identical to existing /api/v1/audit semantics — an admin reading the audit viewer already sees the same content; export just makes it portable.

Mitigation: if a step returns sensitive data, either don’t return it (write it to a secret + return the secret name) or scrub the output before completion. There is no engine-side scrubber on the export path.

Filtering

Query Parameter Effect
include Comma-separated entity list. Subset of: org,projects,environments,audit_events
from RFC 3339 timestamp; audit events older than this are excluded
to RFC 3339 timestamp; audit events newer than this are excluded (clamped to snapshot time)
Terminal window
# Just the audit trail from the last 7 days.
curl -N -H "Authorization: Bearer $KEY" \
  "https://your-cluster.ironflow.cloud/api/v1/export?include=audit_events&from=$(date -u -d '7 days ago' +%FT%TZ)"


# Just the structural meta — no audit rows.
curl -N -H "Authorization: Bearer $KEY" \
  "https://your-cluster.ironflow.cloud/api/v1/export?include=org,projects,environments"

Stream shape

Every line is a complete JSON object. The stream is always:

{"type":"manifest", ...}
{"type":"record", "entity":"...", "data":{...}}
{"type":"record", ...}
...
{"type":"heartbeat", "ts":"..."}        ← every 30s while idle
...
{"type":"footer", "completed_at":"...", "counts":{...}, "body_sha256":"..."}

Manifest (first line, always)

{
  "type": "manifest",
  "version": 1,
  "org_id": "org_acme",
  "started_at": "2026-05-22T14:00:00.000Z",
  "include": ["org", "projects", "environments", "audit_events"],
  "filters": {"from": "", "to": ""}
}

Records

{"type": "record", "entity": "org",          "data": {"id": "org_acme", "name": "Acme Corp", ...}}
{"type": "record", "entity": "project",      "data": {"id": "proj_acme_default", ...}}
{"type": "record", "entity": "environment",  "data": {"id": "env_prod", ...}}
{"type": "record", "entity": "audit_event",  "data": {"id": "evt_01...", "event_type": "run.created", ...}}
{
  "type": "footer",
  "completed_at": "2026-05-22T14:00:02.117Z",
  "counts": {"org": 1, "projects": 3, "environments": 7, "audit_events": 12384},
  "body_sha256": "9f8e2c1b...hex"
}

The body_sha256 covers every byte streamed BEFORE the footer line (manifest + records + heartbeats; the footer itself is excluded). You can verify integrity by re-hashing the same bytes and comparing.

Error envelope (last line on failure)

{"type": "error", "msg": "audit_events: ...", "completed_at": "..."}

A 200 OK status was already on the wire when streaming began. The absence of a footer line — or the presence of an error line — means the export failed and any partial output should be discarded. Counts and hashes are unreliable past the error point.

Snapshot semantics

The stream is bounded to audit_events.created_at <= manifest.started_at. An audit row written DURING the export does not appear in the snapshot. The org.export audit row that records the export itself is written AFTER the stream completes, so consecutive exports of an idle org differ only by N extra org.export rows in the later one — there is no recursive chain.

User-supplied to= is silently clamped at manifest.started_at. Values equal to or after started_at end up at started_at. Use from= to narrow the lower bound; to= only narrows the upper bound, never widens it.

Record order

audit_event records stream newest-first (created_at DESC, id DESC), matching the audit viewer. Other entities (org / project / environment) stream in the underlying store’s natural order. If you need chronological replay, sort the audit subset post-receive.

Timestamp format

Manifest, footer, heartbeat, and error timestamps are RFC 3339 with nanosecond precision (time.RFC3339Nano). The from / to query parameters accept either second- or nanosecond-precision RFC 3339, so you can round-trip manifest.started_at from one export directly into ?to= on the next without truncation.

Status codes

Code When
200 + manifest + footer Success
200 + manifest + error line Mid-stream failure (no footer)
400 Invalid include, from, or to parameter
401 Missing/invalid auth subject
403 Caller lacks org:export (RBAC gate)
429 Another export in flight on this node (Retry-After: 30)

No 5xx is emitted after streaming begins — store errors surface as in-band {"type":"error",...} lines.

Response headers

  • Content-Type: application/x-ndjson
  • X-Accel-Buffering: no (disables nginx response buffering so heartbeats reach the client)
  • Retry-After: 30 on 429 only

Single-flight per node

The endpoint allows one concurrent export per engine node. A second request while another is in-flight returns:

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Multi-node clusters allow one stream per node since each replica serves a different audit + tenancy view. Cluster-wide uniqueness is not a contract; if you need a strictly serialized export, gate it on your side.

Audit trail

Every successful export emits an org.export audit row whose payload carries:

  • actor_id — the API key or user that initiated the export
  • started_at, completed_at
  • include, from, to
  • record_counts — same shape as the footer
  • body_sha256 — same hash as the footer, so you can later prove an archive matches what was streamed
  • bytes_streamed

Failed exports are logged on the cluster (log.Warn) but do NOT produce an org.export audit row.

Operational notes

  • Long streams. Large orgs can take minutes. Configure your HTTP client with no read timeout (curl -N, Node axios timeout: 0).
  • Disk usage. A typical org of ~50k audit rows produces a ~30-50 MB NDJSON file. Plan storage accordingly.
  • Proxy timeouts. If you front your cluster with nginx or a CDN, raise proxy_read_timeout to at least 90s. The heartbeat ticker emits every 30s; a default 60s read timeout will idle-disconnect on the heartbeat gap.
  • Resumption. Not supported in v1. A disconnect mid-stream means you restart the export. Future work: signed-URL async exports.
  • Retries on a flaky connection. The endpoint is GET but writes one org.export audit row on success. Naive HTTP retry middleware will produce duplicate audit rows. Gate retries on terminator detection: only retry when the response body has neither a footer nor an error final line. Or use a client-side idempotency key.
  • Single-flight wedge ceiling. Per-line writes carry a 60s deadline and the heartbeat stop wait caps at 2× that. A stuck slow client tears down the export within ~2 minutes worst case; the single-flight slot releases at that point. The endpoint is not invincible — operators monitoring 429 rate spikes should still investigate.
  • Snapshot equivalence. Two exports taken seconds apart of an idle org should differ only by org.export rows. If they diverge elsewhere, your cluster has uncaptured mutations — file a bug.
  • ADR-0029 — Decision record for this endpoint
  • audit-logging.mdx — How audit events are captured and retained