ironflow cloud

Operator-only. These commands are for the operator of an Ironflow Cloud control plane. They are not for self-hosted users or tenant operators. Ironflow Cloud is planned and no Cloud service runs today. The ironflow cloud commands below read from the control-plane Postgres directly (IRONFLOW_DATABASE_URL) or mint local signing artifacts. The separate ironflow-cloud binary also provides remote inspection through Cloud APIs.

ironflow cloud <subcommand> [flags]

Subcommands:

Subcommand	Description
provision-status	Show current status of a provisioning saga
cap-token mint	Mint an Ed25519 capability token for deprovision saga events
break-glass	Decrypt a break-glass bearer blob for a cluster
snapshot decrypt	Decrypt a snapshot export blob for a cluster
rotation reset	Clear a stuck rotation_in_progress flag for one (cluster, token-kind) row

ironflow-cloud inspect <command>

This is a separate binary built from cmd/cloud, not an ironflow cloud subcommand. Inspection calls the existing authenticated FounderService APIs. It needs no database, engine, worker, or provider credentials on the caller’s machine and does not start a server or worker.

ironflow-cloud inspect <command> --endpoint <origin> --environment <environment> --credential-file <path> [flags]

Command	API and scope	Command flags
health	GetHealthOverview, fleet totals	None
customers	ListCustomers, one page	--cursor, --page-size
customer	GetCustomerDetail, customer, clusters, and up to 20 recent saga runs	Required --customer-id
cluster	GetCustomerDetail, selects one cluster belonging to the customer	Required --customer-id, --cluster-id
operations	ListSagaRuns, one page	--customer-id, --status, --cursor, --page-size
snapshots	ListSnapshotHealth, one page of active or suspended clusters	--stale-only, --cursor, --page-size

Common flags:

Flag	Default	Meaning
--endpoint	Required	Cloud API origin. No credentials, path other than /, query, or fragment. Redirects are refused.
--environment	Required	development, staging, or production. Sent in Ironflow-Cloud-Environment; the server verifies its deployment identity and credential binding.
--credential-file	Required	Owner-only regular file containing an ifstaff_ credential. Symlinks, group/world access, and founder keys are rejected.
--json	false	Protobuf JSON using snake_case fields, including empty fields.
--timeout	15s	Total request deadline, greater than zero and at most 2m.
--help		Print help without connecting.

HTTPS is required except for loopback HTTP with --environment development. The server must set IRONFLOW_CLOUD_ENVIRONMENT to its deployment identity. An unset identity denies staff access. Credentials are not read from environment variables and cannot be supplied as flag values. Use a protected credential file owned by the caller, normally mode 0600, in a private directory.

Staff credentials require a named enabled operator, an explicit fleet_read grant, and the exact relevant permission. customers requires cloud.customers.read; customer and cluster require both cloud.customers.read and cloud.sagas.read; operations requires cloud.sagas.read; health and snapshots require cloud.health.read. Target-constrained and unknown grants are rejected, not widened to fleet access. Every request checks expiry, revocation, and operator state in PostgreSQL. Credentials expire within 24 hours. All Founder mutation RPCs reject these credentials, including direct API callers. No credential administration RPC is available to them. Legacy ifplatform_ founder keys are refused by inspection. Responses contain customer information and must be handled as staff data.

Paged commands default to 50 rows and accept --page-size 1..100, matching the service’s clamp. Each invocation fetches exactly one page and exposes next_cursor. Pass a nonempty cursor back with the same filters to continue. Customer details include only the server’s 20 most recent saga runs; use operations --customer-id for older history. Operations accept status values running, completed, failed, and compensated.

Server output uses an explicit field allowlist and omits billing identifiers. Both output modes clear saga error_message and snapshot last_error, including nested recent saga runs. Arbitrary error text can contain secrets. IDs and cursors remain intact. Human output includes the command, environment, endpoint, and indented data. --json emits only the response object. Data goes to stdout; fixed, sanitized errors go to stderr. Snapshot freshness does not verify recovery.

Exit code	Meaning
0	Success or help
2	Invalid command, flags, arguments, target, or API argument rejection
3	Missing or invalid credential, or unauthenticated response
4	Permission denied
5	Network failure, unavailable endpoint, timeout, or refused redirect
6	Customer or cluster not found
7	Other server, protocol, encoding, or output failure

# Obtain a protected staff credential file through your recovery administrator.
ironflow-cloud inspect health --endpoint http://localhost:8080 --environment development --credential-file ~/.config/ironflow/staff.key
ironflow-cloud inspect operations --endpoint http://localhost:8080 --environment development --credential-file ~/.config/ironflow/staff.key --customer-id cust_example --status failed --page-size 20 --json

Bare ironflow-cloud still starts the control-plane server. Top-level --help prints help; unsupported flags fail before startup. Local founder mint-key bootstrap remains separate from remote inspection.

ironflow-cloud staff-recovery

This separate local recovery boundary requires privileged control-plane database access via IRONFLOW_CLOUD_DATABASE_URL and migration 020. It never starts the server. Ordinary inspection needs neither database nor provider credentials.

All actions require --actor-id (stable named recovery responder) and --reason-ref (incident/change identifier, not free text). The audit records these plus the actual PostgreSQL role, action, target ID, and server time in the same transaction as the change. Recovery administrators have full read delegation authority. These are human credentials; service identity issuance is unsupported.

Action	Required additional flags	Effect
issue	--operator-id, --operator-name, --label, --environment, --permissions, --scope, --ttl, --credential-file	Issue a separately identified key with comma-separated exact read permissions and TTL greater than zero, at most 24h. --scope is fleet_read or targets. A cloud.clusters.provision grant uses --scope targets with --customer-ids (comma-separated customer IDs). Write a new mode-0600 file; print only credential ID.
revoke	--credential-id	Retain revocation timestamp; next request fails.
disable	--operator-id	Disable the named operator and all their credentials on the next request.

IDs, names, labels, and reason references accept 1–128 characters from letters, digits, _ . : @ / -, starting with a letter or digit. Credential identity and authority are immutable. There is no re-enable or in-place permission edit command. Rotation is a recovery procedure: issue a replacement in the same environment with the same or narrower permissions, deliver its file securely, then revoke the previous ID. An ordinary credential cannot rotate itself. Recovery exits 0 on success and 1 on failure. If file delivery fails after issuance, revoke the credential ID reported by the error. Reconcile uncertain recovery outcomes with the audit before retrying.

ironflow cloud provision-status

Read the clusters row and the most recent provisioning_audit rows for a cluster. Connects directly to the control-plane Postgres via IRONFLOW_DATABASE_URL.

ironflow cloud provision-status <cluster_id> [flags]

Arguments:

Argument	Required	Description
cluster_id	Yes	The cluster ID to query (e.g., clu_hooli)

Flags:

Flag	Short	Type	Default	Description
--json		bool	false	Output as JSON
--audit-limit		int	20	Max provisioning_audit rows to show (capped at 1000)

Requires IRONFLOW_DATABASE_URL to be set.

Examples:

ironflow cloud provision-status clu_hooli
ironflow cloud provision-status clu_hooli --json
ironflow cloud provision-status clu_hooli --audit-limit 50

ironflow cloud cap-token mint

Mint an Ed25519-signed capability token that gates the cloud.deprovision-cluster.requested NATS event. The Ed25519 private key is loaded from IRONFLOW_CAP_TOKEN_PRIVATE_KEY_B64 (base64-encoded PKCS8 DER).

ironflow cloud cap-token mint <cluster_id> <customer_id> [flags]

Arguments:

Argument	Required	Description
cluster_id	Yes	Cluster ID (e.g., clu_hooli)
customer_id	Yes	Customer ID (e.g., cust_01h2g3k4j5n6p7q8r9s0)

Flags:

Flag	Short	Type	Default	Description
--ttl		duration	10m	Token time-to-live (max 10m; a longer TTL is rejected at mint time)

Requires IRONFLOW_CAP_TOKEN_PRIVATE_KEY_B64 to be set.

Examples:

ironflow cloud cap-token mint clu_hooli cust_01h2g3k4j5n6p7q8r9s0
ironflow cloud cap-token mint clu_hooli cust_01h2g3k4j5n6p7q8r9s0 --ttl 5m

ironflow cloud break-glass

Decrypt a break-glass bearer blob fetched from the control-plane break-glass S3 bucket. The object key is break-glass/<customer_id>/<cluster_id>. The 32-byte control-plane KEK (the value the saga worker reads from IRONFLOW_CLOUD_KEK_B64) is loaded from IRONFLOW_MASTER_KEY_B64 (base64-encoded). The blob bytes are read from --blob-file (path) or stdin (default). The decrypted plaintext bearer is written to stdout with no trailing newline.

The blob authenticates both identities, so both are required. A cluster slug can be reused by another customer: take the customer ID from the record you are recovering, not from the current owner of the slug.

ironflow cloud break-glass <customer_id> <cluster_id> [flags]

Arguments:

Argument	Required	Description
customer_id	Yes	Customer ID that owned the cluster (e.g., cust_0123456789abcdef)
cluster_id	Yes	Cluster ID (e.g., clu_hooli)

Flags:

Flag	Short	Type	Default	Description
--blob-file		string		Path to encrypted blob (default: stdin)

Requires IRONFLOW_MASTER_KEY_B64 to be set.

Examples:

# Fetch + decrypt via stdin pipe (preferred — blob never touches disk)
aws s3 cp s3://<break-glass-bucket>/break-glass/cust_0123456789abcdef/clu_hooli - \
  | ironflow cloud break-glass cust_0123456789abcdef clu_hooli

# Decrypt a previously-saved blob file
ironflow cloud break-glass cust_0123456789abcdef clu_hooli --blob-file /tmp/clu_hooli.bin

The break-glass bucket is versioned. After deprovisioning, the current object is a delete marker and aws s3 cp returns 404. List versions with aws s3api list-object-versions --prefix break-glass/<customer_id>/<cluster_id>, then fetch one with aws s3api get-object --version-id <id>.

ironflow cloud snapshot decrypt

Decode the historical single-tag snapshot format. Current Cloud backups use IFS1 complete generations and require ironflow-cloud-recover, described below. This command cannot decode those generations. The 32-byte control-plane KEK is loaded from IRONFLOW_MASTER_KEY_B64 (base64-encoded). The blob bytes are read from --blob-file (path) or stdin (default). The decrypted plaintext (typically a gzipped pg_dump) is written to stdout.

ironflow cloud snapshot decrypt <cluster_id> [flags]

Arguments:

Argument	Required	Description
cluster_id	Yes	Cluster ID (e.g., clu_hooli)

Flags:

Flag	Short	Type	Default	Description
--blob-file		string		Path to encrypted blob (default: stdin)
--max-bytes		int64	268435456 (256 MiB)	Maximum blob bytes to read (held in RAM during decrypt; peak memory ≈ 2× blob size)

Requires IRONFLOW_MASTER_KEY_B64 to be set.

Examples:

# Fetch + decrypt via stdin pipe
aws s3 cp s3://ironflow-snapshots-fsn/exports/clu_hooli/run_xyz.tar.gz.enc - \
  | ironflow cloud snapshot decrypt clu_hooli > dump.tar.gz

# Decrypt a previously-saved blob file with a 4 GiB cap
ironflow cloud snapshot decrypt clu_hooli \
  --blob-file /tmp/clu_hooli-export.enc \
  --max-bytes 4294967296 > dump.tar.gz

ironflow-cloud-recover

Authenticate and validate a complete Cloud backup generation from object storage. Build this standalone operator client from the matching source with make ironflow-cloud-recover. Linux engine release archives include the engine and ironflow-backup. Cloud remains pre-launch.

ironflow-cloud-recover --cluster-id clu_example --run-id RUN_ID \
  --object-key EXACT_STORED_KEY --output /data/recovery/generation.tar.gz \
  --staging-dir /data/recovery

The output must not already exist. Use a private directory on encrypted storage; the archive includes recovery keys. --verify-only validates without creating the output file and requires --staging-dir. Omitting --object-key selects the customer export key derived from the cluster and run IDs. Daily meta generations require their exact stored key and matching generation identity.

Set IRONFLOW_CLOUD_BREAK_GLASS_ENDPOINT, IRONFLOW_CLOUD_BREAK_GLASS_BUCKET, IRONFLOW_CLOUD_BREAK_GLASS_ACCESS_KEY, IRONFLOW_CLOUD_BREAK_GLASS_SECRET_KEY and IRONFLOW_CLOUD_KEK_B64. The KEK must decode to 32 bytes and remain available independently of the failed host. IRONFLOW_CLOUD_BREAK_GLASS_REGION selects the S3 region. IRONFLOW_CLOUD_BREAK_GLASS_SECURE defaults to true.

The client rejects PostgreSQL-only exports and unsupported archives. It emits no plaintext to stdout. Apply a validated archive with ironflow-backup restore --archive ARCHIVE --target TARGET_JSON on a prepared isolated host. Inspection and restore accept --staging-dir, defaulting to /data/ironflow-backup. Restore requires explicit empty component roots, empty databases, matching local roles and stopped services. It leaves activation and recovered-data checks to the operator. See ADR 0159 for the generation contract.

ironflow cloud rotation reset

Clear a stuck cloud.cluster_tokens.rotation_in_progress flag for one (cluster_id, kind) row. The boot sweep clears rows whose rotation_started_at exceeds the stale threshold automatically (#1074 PR2); this CLI is for the case where an operator has inspected a partial-rotation state and wants to drop the gate before the sweep window elapses. The action is audited via cloud.token_rotation_audit (result=success, step_up_method=operator_cli); the cluster’s token value_hash and meta_push_key are not touched — this is purely a flag clear. Connects directly to the control-plane Postgres via IRONFLOW_DATABASE_URL.

ironflow cloud rotation reset --cluster <cluster_id> --kind <kind> [flags]

Flags:

Flag	Short	Type	Default	Description
--cluster		string		Cluster ID (clu_*) (required)
--kind		string		Token kind: engine_api_key, founder_api_key, or dashboard_jwt_secret (required)
--reason		string		Operator note recorded in the audit row

Requires IRONFLOW_DATABASE_URL to be set.

Examples:

ironflow cloud rotation reset --cluster clu_hooli --kind engine_api_key --reason "manual recovery after PG failover"

---

ironflow-cloud provision

Staff-assisted provisioning requires a customer-scoped cloud.clusters.provision grant. Cloud remains planned and is not running. This command requires a configured Cloud API and durable provision worker. Resize, deprovision, restore, and terminal replacement are not available through this command.

Every request requires --endpoint, --environment, and an owner-only --credential-file. HTTPS is required except for loopback development. Credentials are never accepted as flag values. Output is allowlisted JSON. --timeout defaults to 15 seconds and must be positive and at most two minutes.

Preparation requires --prepare, --customer-id, --slug, --tier, and --region. It returns preparation_id, the validated intent, and expires_at. Preparations expire after ten minutes and authorize only a fresh, never-claimed target.

Submission repeats the same customer, slug, tier, and region and adds --preparation-id, --request-key, and --reason-ref. The reason is a ticket reference justifying staff-assisted onboarding and billing bypass. Retain the exact submission after a lost response. An identical authorized retry returns the same operation. A changed intent under the same key fails. Credential rotation preserves request identity when the replacement has the same operator and sufficient scope.

A successful submission reports durable acceptance, not provisioning completion. Claims remain occupied after failure or cleanup. Use a fresh slug for another provisioning operation. Ordinary staff cannot release claims.

ironflow-cloud inspect operation

--operation-id ID selects one admitted staff provision operation. The endpoint, environment, credential file, and timeout rules above apply. The credential must currently authorize provision for the operation’s customer and environment.

JSON reports operation_id, customer, slug, tier, region, environment, admission time, run ID, state, and effect records. States are queued, running, succeeded, failed, cancelled, stale, and compensation_incomplete. Effect records contain only step, outcome, and timestamp. Raw provider errors and credentials are omitted. inspect operations remains the separate legacy saga-list command.

The provision and single-operation commands return exit 2 for invalid arguments, 3 for an invalid credential file, 4 for a rejected or unavailable request, and 1 for output failure. Exit 0 means the requested response was received.