Debug a surprising deny

Someone gets a 403 that they did not expect. The cause is often Layer 1 RBAC. The next most common cause is the policy set. This guide shows how to tell them apart and how to find the rule.

For the model, read Authorization policy sets.

The symptom

The caller sees:

HTTP 403 Forbidden
{ "error": "insufficient permissions", "code": "permission_denied" }

The body is the same for a Layer 1 deny and a Layer 2 deny. It does not carry the reasons, because reasons can contain request data. The reasons are in the audit row.

Step 1. Find the audit row

Ironflow always writes a denied decision to the auth audit log. List them with the audit API. Filter by the action or by the key of the caller:

curl -s -H "Authorization: Bearer $IRONFLOW_API_KEY" \
  "http://localhost:9123/api/v1/audit?event_type=authz.decision.denied&action=runs:delete&api_key_id=ak_1234&limit=20"

A row from the policy set looks like this (fields trimmed):

{
  "api_key_id": "ak_1234",
  "action": "runs:delete",
  "resource": "irn:ironflow:org_acme:proj_default:run:env_default:run_1",
  "decision": "denied",
  "environment": "env_default",
  "method": "DELETE",
  "path": "/api/v1/runs/{id}",
  "reason": "members cannot delete runs",
  "policy_set_version": 4,
  "policy_set_mode": "enforce"
}

Read the row like this:

• path is the route label for new HTTP decisions. Older rows can retain concrete normalized paths. The target identifier remains in resource where supplied.
• policy_set_version and policy_set_mode are present. Layer 2 was consulted, and Layer 2 denied. reason holds the sorted reasons of the set, joined with ; . The text is cut at 1 KiB, with a …(+N more) suffix.
• Both fields are absent and the reason is insufficient permissions. Layer 1 RBAC denied. Look at the roles of the caller, not at the policy set.
• The reason is <rule path> is false or <rule path> is undefined. The decision rule of the set denied. For example: data.envoy.authz.allow is false.
• The reason is policy evaluation error (<class>). The set could not decide, and the engine failed closed. See Error classes.
• The reason is non-string deny reason. A deny member was not a string. The request is still denied.
• The reason is organization not resolved. Policy enforcement is on, and the environment of the webhook source has no project, or the project has no organization. No policy was evaluated. Fix the environment, not the set.
• The row has webhook_source_id. The subject was that webhook source, not a key or a user, so api_key_id is empty.

If the organization has no set, there is no policy_set_version and Layer 2 has no opinion.

Step 2. Get the version that decided

The row names the version. List the versions:

ironflow policy versions

To read the files of one version, use the dashboard: Policies, then Version history, which shows a diff per file. Or use the REST route:

curl -s -H "Authorization: Bearer $IRONFLOW_API_KEY" \
  http://localhost:9123/api/v1/policy-set/versions/3 | jq -r '.modules | to_entries[] | "# " + .key + "\n" + .value'

Compare the version of the row with the active version. A later version may already fix the rule.

Step 3. Reproduce with a dry-run

Build an input from the row. Use the action and resource of the row, and the subject as it was at the time. Roles can change after the row was written, so check the roles that the caller had then.

{
  "request": { "action": "runs:delete", "resource": "irn:ironflow:org_acme:proj_default:run:env_default:run_1", "environment": "env_default", "org_id": "org_acme" },
  "subject": { "id": "", "api_key_id": "ak_1234", "roles": ["member"], "user_email": "" }
}

Dry-run the active set, or a directory with your edits:

ironflow policy dry-run --input request.json
ironflow policy dry-run ./policies --input request.json --explain

The result shows decision, the denied by entry point (deny, the decision rule, or both), the reasons, and an evaluation error line with the class if the set failed. With a directory, the dry-run uses the files on disk, and the CLI reports the set mode as monitor. The decision field is the verdict of the set. It is deny even for a set in monitor mode.

To dry-run the decision rule, add the Envoy view to the input (an attributes key). Use the actual request path and query from the caller, not the audit route label. The audit row cannot reconstruct that input. Without the Envoy view the dry-run evaluates deny only, as a handler-side check does.

Explain and trace

--explain prints OPA trace notes for the input. Add trace calls to the rules that you doubt. A trace call writes a note and succeeds:

package ironflow.authz

deny contains "members cannot delete runs" if {
  "member" in input.subject.roles
  trace(sprintf("action is %s", [input.request.action]))
  input.request.action == "runs:delete"
}

Run the dry-run with --explain. The notes show which rules ran and which lines they reached. A rule that stops before your trace call is not firing. Remove the trace calls before you push. print() also works in a dry-run. Its output goes to the engine debug log with the organization ID. It never goes to a response or an audit row, and live decisions do not print.

Error classes

The field error of a dry-run, and the class in policy evaluation error (<class>) of an audit row, name why the set failed:

Class	Cause	Fix
builtin_error	A builtin got a bad value, for example startswith(123, "a").	Check the type before the call, or fix the data path.
conflict	A complete rule produced two different values for one input.	Make the rule single-valued, or add else.
timeout	The set ran longer than 50 ms.	Simplify the rule. Avoid big loops over lists in input.
panic	The evaluator failed with an internal error.	Report it with the input. The request was denied.
eval_error	Another error from OPA.	Read the message in the dry-run.
compile_error	The stored set no longer compiles on this node, for example after an OPA upgrade.	Run ironflow policy validate, fix the files and push. See Recover from a lockout.
load_error	The node could not read the set from the store and has no earlier copy.	Check the store. A node that already loaded the set keeps serving it.

In monitor mode these cases are would_deny rows. They block nothing.

Time windows

A rule can use the time of the request. This rule denies during a change freeze:

package ironflow.authz

deny contains "change freeze" if {
  now := time.now_ns()
  now >= time.parse_rfc3339_ns("2026-12-20T00:00:00Z")
  now < time.parse_rfc3339_ns("2027-01-05T00:00:00Z")
}

The dry-run uses the real time. To test a window, test the rule with opa test and fix the clock in the test, or dry-run before and after the window.

Common reasons for a mismatch

If the dry-run disagrees with the audit row, work through this list:

1. A different version. The row names the version that decided. The dry-run uses the active set.
2. The subject changed. Roles, email and groups are those at the time of the request. Copy them from the time of the row, not from now.
3. An API key has no email. subject.user_email is empty for a raw API key without a bound user. A rule that tests the email behaves in a different way for keys.
4. The environment. The row has the environment of the request. Use it in request.environment.
5. The gate and the handler. Some actions are checked twice. The gate sees the route and an empty resource. The handler sees the real IRN. Dry-run both. See Limit an agent key to specific functions.
6. Another node has the old set. After a write, each node adopts the new version. The gauge ironflow_policy_set_version{org} shows the version on each node. A 503 policy_set_not_propagated on the write means some nodes may still run the old version. See Policy set errors.

Step 4. Fix it

• The rule is wrong. Edit the file and push. The new version appears in later rows.
• The rule is right and the request is wrong. Fix the caller.
• The rule was a mistake. Roll back. See Author a policy set.
• You are not sure. Switch the set to monitor mode and read the rows. See Roll out with monitor mode.
• The rule blocks policy writes for your admins. See Recover from a lockout.

What dry-run does not test

• Layer 1. A dry-run evaluates the policy set only. If RBAC denies, the set never runs.
• Mode. A dry-run reports the verdict of the set, also for a monitor set. It does not tell you whether the request would be blocked.
• Other orgs. A dry-run uses the set of your organization.