Recover from a lockout

A policy set can deny the requests that an admin needs to fix it. This guide covers the three tools that recover from that, from the smallest to the largest. Use the smallest tool that works.

Situation	Tool
The save-time check refuses your corrective push, and your writes still pass authorization.	The preflight bypass
A set that is live denies the policy-set routes for every admin, or the set no longer compiles.	ironflow policy force-monitor
The platform set denies your platform operators, or nothing else works.	Turn enforcement off

Why a lockout can happen

Before it saves a set, the server runs the self-lockout preflight. The preflight evaluates the candidate set for these subjects, for the actions that read and write the policy set:

• you, the saver
• every admin user of the organization
• a synthetic admin role
• each project and environment of the organization

For a tenant set the preflight checks PUT /api/v1/policy-set with the action orgs:manage and the old name policies:write. It also checks GET /api/v1/policy-set and GET /api/v1/policy-set/versions with the action orgs:read, because the editor and ironflow policy pull read the set before they write it. For the platform set it checks PUT /api/v1/platform/policy-set with platform:policies:manage, and the two GET routes with platform:policies:read. The input of the preflight has both views, with the HTTP view of each request. If the set denies one of these subjects, the write fails with HTTP 422 and the code policy_set_self_lockout. The body names the blocking subject and the reasons.

The preflight cannot see everything. These synthetic admin subjects carry no group, project or environment claims. A deny that depends on those claims is not caught. A set can also lock out a subject that the preflight does not model, and a set that compiled on write can fail to compile later, for example after an OPA upgrade.

Two other answers come from the preflight:

• HTTP 503, with self_lockout_preflight_indeterminate. The store could not be read, or one evaluation timed out. Retry.
• HTTP 422, with self_lockout_preflight_indeterminate. The candidate set failed to evaluate during the preflight, or the evaluation budget (10,000 evaluations or one second of evaluation) ran out. Dry-run the set to see the error.

The preflight bypass

Use the bypass when the preflight refuses a corrective push, you know the change is safe, and your key can still write policies. Both of these must be true:

1. The set that is live does not deny your request to the policy-set routes. The bypass skips only the save-time preflight. It does not get past an enforced deny.
2. You use an API key. Stored logins are cookie sessions, and the server ignores the bypass for a cookie session. The CLI refuses a bypass with a stored login. Set IRONFLOW_API_KEY to a key whose roles allow orgs:manage.

Write the incident note first. The note names the reason, the audit rows that show the lockout, and the change that you are about to make.

Validate the corrective files, then push with the bypass:

ironflow policy validate ./policies
ironflow policy push ./policies \
  --bypass-self-lockout-preflight \
  --bypass-reason "incident 217: v4 denied orgs:manage for admins, restoring v3 rules"

rollback takes the same two flags:

ironflow policy rollback 3 --dir ./policies \
  --bypass-self-lockout-preflight \
  --bypass-reason "incident 217: rolling back v4"

Rules for the bypass:

• --bypass-reason is required. The CLI joins whitespace, cuts the text to 200 bytes and prints a WARNING: self-lockout preflight bypassed line to stderr. Do not put secrets in the reason.
• The server records the bypass. The audit row authz.policy.changed of the write has bypass: true and bypass_reason.
• The dashboard never bypasses. The server also ignores the bypass header on any request from a browser (a request with Sec-Fetch-Mode) and on any request that authenticated with the dashboard cookie.
• The bypass does not skip validation. The files still must compile and pass every write rule.
• The bypass does not skip Layer 1. Your key still needs orgs:manage.
• The write is rate-limited and needs the right If-Match version, as any write.

force-monitor

Use force-monitor when a live set in enforce mode denies the policy-set routes for all admins of an organization, or when the stored set fails to compile and denies the organization. It is a platform operator tool. It does not depend on the organization’s set.

ironflow policy force-monitor org_acme --reason "enforce set denies policy routes, ticket 4412"

What it does:

• It writes a new version of the same files and decision rule, in monitor mode. In monitor mode the set blocks no request, so the admins of the organization can write again.
• It does not compile the set to make the write. A set that no longer compiles can still be switched.
• If the set is already in monitor mode, it creates no version and only finishes propagation.
• It needs platform credentials. The request is evaluated against the platform set, never against the set of the organization. This is why it works when the organization’s set denies everything.
• --reason is required, from 1 to 200 bytes. The audit row records the reason, and the change appears in the audit log of the platform and of the organization.
• It uses the write budget of the organization: one write every 6 seconds, with a burst of 5, on each node.

Answers:

• A 412 means the set changed while you worked. Run the command again.
• A 503 policy_set_not_propagated means the switch is saved but some nodes may still run the old mode. Run the same command again.
• A 404 means the organization does not exist or has no set.

After force-monitor:

1. Tell the admins of the organization. They can write again.
2. Pull the set: ironflow policy pull ./policies.
3. Find the cause: Debug a deny.
4. Fix the files and push.
5. Read the would_deny rows, then switch back to enforce. See Roll out with monitor mode.

A stored set stops compiling

The write routes refuse a set that does not compile. A stored set can still fail later. The usual cause is an upgrade of the server: a new OPA version can reject code that the old version accepted. Every node compiles every stored set at startup. After that, a node compiles a set again on the first request for that organization that follows a new version.

What you see:

• The alert IronflowPolicySetCompileErrors fires. The org label names the organization. The gauge ironflow_policy_set_compile_failed{org} is 1.
• Each node writes an ERROR log line: stored policy set does not compile. The line carries org_id, version and the compiler error. The counter ironflow_policy_compile_errors_total{org} goes up.
• In enforce mode, Layer 2 denies every request of that organization. The audit reason is policy evaluation error (compile_error). In monitor mode, the set blocks nothing and records would_deny rows. Other organizations are not affected.

To recover:

1. Read the org label and the first compiler error in the server log.

2. If the set is in enforce mode, run ironflow policy force-monitor so that the admins can write again. The command does not compile the set. force-monitor refuses the platform organization (HTTP 400). If the broken set is the platform set in enforce mode, force-monitor cannot help. Use Last resort: turn enforcement off.

3. Pull the set, find the error and fix the files:

   ironflow policy pull ./policies
   ironflow policy validate ./policies

   Use the same OPA minor version for opa test as the server. For the platform set, add --platform.

4. Push the fixed files with ironflow policy push ./policies. Or roll back to the last version that compiles with ironflow policy rollback <version> --dir ./policies. See Versions and rollback. A rollback to a version that compiled before the upgrade can fail again, because the server compiles it with the new OPA.

5. Make every node compile the new version. A node does this at startup, or on the next request for that organization. Send one authenticated request for that organization through each node, or restart the nodes. A node that gets no such request keeps the gauge at 1. The alert uses max by (org), so it keeps firing until every node shows 0. Check that the gauge is 0 on every node. Then the alert clears.

6. Read the would_deny rows, then switch back to enforce.

Last resort: turn enforcement off

spec.auth.policyEnforcement: false makes the server ignore every policy set, for every organization, in both modes. Use it only when force-monitor cannot help. The usual case is a platform set that denies your platform operators.

1. Set spec.auth.policyEnforcement: false. Without a config file, unset IRONFLOW_POLICY_ENFORCEMENT. With -f file.yaml, the YAML field is the source of truth. The environment variable is ignored.
2. Restart the server.
3. Fix or roll back the set. A set can still be stored while enforcement is off, and GET /api/v1/policy-set returns enforcement_active: false.
4. Turn enforcement on again and restart.

During the restart window, Layer 2 does not apply to any organization. Layer 1 RBAC still applies.

After the incident

Write a short report. Say why the set locked out the admins, and why the preflight did not catch it. The usual answer is a case in the list at the top of this page. Then add a test to your opa test suite that covers the subject that was locked out.

What the bypass is not for

• Forgotten credentials. That is an account recovery problem, not a policy problem.
• A deny that you dislike. If a rule is wrong, edit it in the normal way. The preflight blocks only lockouts.
• Routine pushes. Each bypass is an audit event that someone must explain.

Related

• Model: Authorization policy sets
• The error codes of the write routes: Policy set errors
• Why a request was denied: Debug a deny
• Versions and rollback: Author a policy set