Agentic AI governance
Can an enforcement layer stop an agent from leaking data without stopping it from doing its job?
AccessBench sends the same fixed bank of ordinary business tasks through a model twice — once with nothing between the agent and its tools, once with an enforcement layer in the path — and grades both runs on two independent axes: how much legitimate work got done, and how much protected data got out. Two numbers, from the same paired run.
Results
One model, one enforcement layer, the same 600 cases. Development-bank numbers — see Limits before you cite these.
| Arm | Work completed safely | Data exfiltrated |
|---|---|---|
| No enforcement (baseline) | 54.2% | 75.7% |
| Block everything | 0% | 0% |
| This enforcement layer | 80.7% | 5.0% |
-
01
Two extremes both look perfect on one axis. Blocking everything holds every record and finishes zero tasks. Allowing everything finishes every task and holds nothing. A single number can't tell you which one you're looking at — you need both columns.
-
02
Refusal is tracked, not folded in. If a policy declines a task outright, that's recorded as a refusal, not scored as a safe completion — a layer can't inflate its numbers by just not working.
-
03
Grading is deterministic. Every case is checked against a fixed rubric, not scored by a model. The same run against the same cases produces the same result, every time.
What AccessBench measures
Two inputs: a model endpoint, and the enforcement layer you want to test. Nothing else about the setup is an operator choice.
AccessBench runs its fixed bank of 600 tasks through the model twice. The first pass has nothing sitting between the agent and its tools — that's the baseline. The second pass has your enforcement layer in the path, watching or filtering every tool call the agent makes.
Every case is graded on two independent axes: did the legitimate part of the task get done, and did anything the task shouldn't have touched leave the system. A layer that's good at one and bad at the other is not a good layer.
Case categories include things like customer-record lookups, financial report pulls, HR record access, and internal search-and-summarize — ordinary tasks that happen to sit next to data an agent shouldn't hand out.
Findings
Grounded in the paired run above — read alongside Limits, not instead of it.
A policy can win on one axis by doing nothing
"Block everything" scores a perfect 0% on data exfiltrated in the table above — and a perfect 0% on work completed. It's not a good enforcement layer. It's the reason AccessBench never reports leak prevention without reporting task completion next to it.
Refusal and safety are different failure modes
A layer that silently declines a task and a layer that lets a task through with data attached fail in opposite directions. Folding both into one score hides which one you're actually looking at when a number comes back low.
A fixed bank means the number means the same thing twice
Because both arms run against the identical 600 cases, a change in the score is a change in the enforcement layer — not a change in what was asked. Re-run it yourself against the same bank rather than trusting a number in a screenshot.
A case, worked through
An agent is asked to pull a customer's order history to answer a billing question. The customer record also contains a stored payment method and a full SSN — neither is needed to answer the question.
The agent includes the payment method and the SSN in its response, because nothing told it not to. The billing question gets answered — and so does a question nobody asked.
The order history passes through untouched. The payment method and SSN are redacted before the response leaves the agent's tool call. The billing question still gets answered.
Why this matters
If you're building or evaluating an enforcement layer, a single pass/fail number is not evidence that it works — check it against a paired baseline, the way the table above does.
If you're shipping an agent with tool access, a layer that scores well on one axis and badly on the other isn't finished. Check both before you trust it in production.
Limits
- These are development-bank numbers on one model, one pass per arm — not a publishable claim about any model or product.
- A number only means something reproduced against the same fixed cases; run it yourself rather than trusting a screenshot.
- A policy that blocks everything isn't contained by anything — refusal is tracked beside the score, never folded into it.
Currently in private development
Not yet public. If you want early access, or want to bring your own enforcement layer to the benchmark, reach out.