Artifact overflow to blob storage (SeaweedFS / S3)

When a step or run returns a large output — or an entity stream writes a large snapshot — Ironflow can transparently move that payload out of the database and into a blob backend, leaving only a small reference behind. Your functions never see the difference — reads inflate the payload back automatically.

This guide shows how to enable overflow and point it at a self-hosted SeaweedFS S3 endpoint (or any S3-compatible backend).

How it works

  large value                         blob backend
  ───────────                         ────────────
  > threshold  ──offload──▶  {"__blob__":true,...}  ──▶  artifacts/{env}/{run}/{step}/v{ver}/output-{hash}    (steps)
                                                        runs/{env}/{run}/v{ver}/output-{hash}                 (run outputs)
                                                        snapshots/{env}/{type}/{id}/v{ver}/state-{hash}       (entity-stream snapshots)
  ≤ threshold  ──inline──▶  stored in the DB column
  any read     ◀──inflate──  full bytes restored before your function sees it

• Threshold-gated. Only outputs larger than the configured threshold offload. Everything else stays inline in the database.
• Transparent. Functions, the dashboard, ironflow inspect, and event subscribers all receive the full output — the reference is resolved server-side.
• Reads always inflate. Disabling offload later (threshold 0) does not break reads of outputs that were offloaded while it was on.

Single-node default: local filesystem

On a single node, overflow needs no backend configuration. Blobs are written to a blobs/ directory next to the SQLite database (the data directory). Overflow is still off until you set a threshold — see Enable overflow.

The local filesystem backend is for single-node deployments only: the files live on one node’s disk and are invisible to other nodes. A multi-node cluster must use a shared S3 backend (Ironflow refuses to start a cluster on the filesystem backend).

Enable overflow

Set a threshold. Outputs larger than it offload; the value accepts human-readable sizes (1MB, 512KiB, 4MB).

ironflow.yaml

spec:
  blobs:
    artifactThreshold: "1MB"   # offload outputs larger than 1 MB

Or via environment variable:

IRONFLOW_ARTIFACT_THRESHOLD=1MB ./ironflow serve

An empty value or 0 disables offload (the default). The read/inflate path is always active regardless. Run and step redaction cleanup also stays active with threshold 0; see Redacted run and step blobs.

Point overflow at SeaweedFS

SeaweedFS exposes an S3-compatible API and is the recommended self-hosted backend.

1. Run SeaweedFS with the S3 gateway

# Single-binary dev/self-host: filer + volume + S3 API on :8333
weed server -dir=/data/seaweedfs -s3 -s3.config=/etc/seaweedfs/s3.json

Set the fsync rule, or a host reset can lose blobs

By default, SeaweedFS acknowledges a write before the write is on disk, and a host reset loses blobs that it acknowledged. Use SeaweedFS 4.49 or later, and set this rule before anything writes to the server:

weed shell <<< "fs.configure -locationPrefix=/buckets/ -fsync=true -apply"
sync
curl http://localhost:8888/etc/seaweedfs/filer.conf

The last command must show "fsync": true on /buckets/. The rule itself is not synced, so without sync a reset soon after can lose it, and later writes then lose blobs with no error. The Starter, Cluster and HA Compose files do this for you. Earlier releases sync only the data file under this rule, so they still lose blobs.

Define an access key/secret and a bucket in the S3 config:

/etc/seaweedfs/s3.json

{
  "identities": [
    {
      "name": "ironflow",
      "credentials": [
        { "accessKey": "ironflow", "secretKey": "REPLACE_WITH_A_SECRET" }
      ],
      "actions": ["Read:ironflow-artifacts", "Write:ironflow-artifacts", "List:ironflow-artifacts"]
    }
  ]
}

Create the bucket once it is running:

weed shell <<< "s3.bucket.create -name ironflow-artifacts"

2. Provide credentials to Ironflow

The S3 driver reads credentials from the standard AWS environment variables:

export AWS_ACCESS_KEY_ID=ironflow
export AWS_SECRET_ACCESS_KEY=REPLACE_WITH_A_SECRET
# AWS_SESSION_TOKEN is also honored if your backend issues temporary creds

Never put credentials in blobs.url or commit them — they belong only in the environment (or your secret manager).

3. Point Ironflow at the endpoint

The blob URL is s3://{bucket}?endpoint={url}. The endpoint scheme selects TLS: https:// connects securely, http:// does not. region is optional. The URL takes no path: s3://bucket/prefix is rejected at startup, because the bucket is the whole host.

ironflow.yaml

spec:
  blobs:
    url: "s3://ironflow-artifacts?endpoint=https://seaweedfs.internal:8333"
    artifactThreshold: "1MB"

Or via environment:

export IRONFLOW_BLOB_URL="s3://ironflow-artifacts?endpoint=https://seaweedfs.internal:8333"
export IRONFLOW_ARTIFACT_THRESHOLD=1MB
./ironflow serve

SeaweedFS uses path-style addressing, which the driver detects automatically for non-AWS endpoints — no extra configuration needed.

Using AWS S3 or another provider instead

The same s3://bucket?endpoint=... URL works for any S3-compatible backend (AWS S3, MinIO, Cloudflare R2, Backblaze B2). Set endpoint to the provider’s S3 endpoint and supply the matching AWS_* credentials.

Files use the same backend

The file storage API stores its bytes in the same blobs.url backend, under the key prefix files/. Do not add a lifecycle rule to the S3 bucket that expires that prefix. A rule like that deletes user files.

On the filesystem backend, file bytes go to the same blobs/ directory, beside the database. The database size limit (HTTP 507) does not count them. This release has no file quota. Monitor that volume, or use an s3:// backend.

Configuration reference

YAML (spec.blobs)	Environment variable	Default	Meaning
url	IRONFLOW_BLOB_URL	(empty → local filesystem)	s3://bucket?endpoint=...&region=... selects the S3 backend
artifactThreshold	IRONFLOW_ARTIFACT_THRESHOLD	(empty → off)	Size above which outputs offload (1MB, 512KiB, …)

Environment variables override the YAML values.

Redacted run and step blobs

Run and step redaction commits a permanent database marker in the same transaction as payload and audit scrubbing. A run marker protects input and output; a step marker protects output and original_output. Empty protected columns stay NULL. Later writes preserve those columns, payload patch attempts conflict, and creating a new row with the same marked identity conflicts even after the old row is deleted. Recognized redaction placeholders are never offloaded.

Each marker authorizes deletion from one exact prefix:

Redaction	Blob prefix
Run	runs/{environmentID}/{runID}/
Step	artifacts/{environmentID}/{runID}/{stepName}/

Run redaction does not sweep step artifacts. Redact each step whose output you need removed. Step redaction does not sweep other steps or the run’s blobs.

Ironflow attempts cleanup immediately after commit, using a bounded context that survives request cancellation. Failure does not roll back committed redaction. A separate server worker retries every permanent marker. It starts whenever the overflow blob backend is wired, including with threshold 0 and with files.reconcile=off or dry-run. These file settings do not make redaction cleanup a dry run. No extra cleanup setting is required.

The first pass starts after up to five seconds of startup jitter. The worker waits one minute after a completed pass before starting another. A pass captures an upper marker boundary and reads pages of at most 1000 markers, the default and maximum page size. Failed prefixes do not block visits to later markers; markers inserted behind the cursor or above the boundary wait for the next pass. Each prefix sweep has a two-minute timeout. The filesystem and S3 backends list at most 1000 objects per page and use batch deletion where supported. Custom backends without InfoLister use the compatibility List fallback, whose memory use depends on the backend.

Writers check markers before offload and again after an upload attempt that may have written bytes, including failed or uncertain uploads and rejected or skipped database writes. A failed marker lookup prevents offload. Blob backend failures still fall back to inline storage, subject to the same database protection. An upload already in flight can finish after an empty or successful sweep, so a transient plaintext object can exist until a later sweep removes it. After the last delayed upload finishes, removal requires the authoritative database, blob backend and worker to remain available and cleanup to make progress. Database redaction and object deletion are not one atomic transaction.

Markers never expire or complete. Successful deletion, an empty listing, partial failure and timeout all leave them intact. They survive run, step, environment, project and organization deletion, and server shutdown interrupts work without removing its marker. Every node may revisit the same markers. This adds permanent database storage and recurring database reads and blob listings, even for empty prefixes. Cleanup load grows with the number of marked identities; the one-minute wait is between completed passes, not a deadline for removing bytes.

Upgrade and recovery limits

Upgrade all writing nodes before relying on protected database fields. Older writers can ignore markers and restore plaintext during a mixed-version rollout. The migration finishes before workers and writes start. It backfills markers from surviving run and step rows whose protected columns contain recognized redaction placeholders, using the same recognition as IsRedacted. A string mentioning $redacted or an explicitly false flag does not qualify.

Invalid legacy prefix identities are skipped rather than used as deletion authority. Components must be nonempty, cannot be . or .., and cannot contain slashes, backslashes, ASCII control characters or DEL. Ordinary writes and patches remain usable for these identities, with new payloads kept inline and no new blob offload or deletion authority. Existing blob references remain readable. Previously redacted rows already deleted before upgrade have no recoverable automatic cleanup authority. Ironflow does not infer redaction from missing rows, absent references or object age.

One authoritative database must own each environment/run/step prefix. Shared identity namespaces across independent databases, including writable database clones, remain unsupported. Keep their blob namespaces separate. The guarantee begins at durable redaction commit and depends on retaining marker history with the authoritative database. Restoring a backup taken before redaction loses its evidence and is outside this guarantee. Redaction does not remove plaintext from older backups.

Other cleanup boundaries

Intermediate-version blobs and failed uploads outside marked prefixes are not collected automatically. Run deletion and environment, project or organization deletion still use best-effort prefix sweeps for unmarked blobs; failed sweeps have no durable retry authority. Event redaction and entity-stream snapshot cleanup have separate behavior and gain no retry guarantee from run or step markers. General orphan and snapshot reclamation remain outside this worker.

File cleanup uses its separate durable retirement queue under files/. It does not collect unexplained crash, temporary or legacy objects merely because they lack references. Its reconciliation modes control that worker only. Run and step redaction never sweep file prefixes. NATS copies, backups and plaintext outside the exact marked prefixes are outside redacted-prefix cleanup.

Operational notes

• Multi-node requires S3. A cluster (external NATS) refuses to start on the filesystem backend — set an s3:// URL. Reads of previously-offloaded outputs need the shared backend too, so the requirement holds even with offload off.
• Deleting an environment reclaims its blobs eagerly. Ironflow sweeps all offloaded blobs for that environment from the blob backend after the database rows are removed. Deleting a project or an organization sweeps the environments underneath it too, but only where the delete itself succeeds: an environment that still holds functions blocks the cascade with a foreign key error, so empty the environment first. The sweep runs detached from the request’s cancellation, so a client disconnect mid-delete does not strand the blobs — though the delete call itself does not return until the sweep finishes, so a very large delete can take a while to respond. This sweep is best-effort for unmarked blobs. Redacted prefixes retain their separate permanent retry markers. Unmarked intermediate versions still accumulate inside a live environment; general orphan collection is not implemented. A bucket lifecycle policy can also delete live offloaded payloads, so account for their read lifetime before enabling one.
• Missing blobs surface explicitly. If an offloaded blob is gone from the backend, the read fails with an explicit “output unavailable” state (a non-retryable DataLoss), naming the missing key — never a silent or opaque error.
• Metrics. When metrics are enabled, overflow records offloads, bytes offloaded, degrades (offload failed → written inline), backend errors, and unavailable (missing-blob) events. See Observability.