Skip to content

Benchmarks

Every published measurement carries the date it was last re-run, and each date is machine-checked: scripts/check_benchmark_freshness.py fails the release if any row is missing its marker, and --release fails if any row is older than 30 days.

Two conventions are worth knowing before reading any number here.

A null result is published as a null result. The multi-harness prompt-injection row is a zero, and the page says at length why that zero is not evidence of injection resistance. A benchmark that only ever reports wins is not a benchmark.

Scope is stated, not implied. Where a comparison is against a system this harness does not execute, the incumbent's coverage is marked scope-claimed, not re-run rather than given a fabricated number. Where a slot has no samples, it is shown as n=0 rather than omitted — four of the ten OWASP Agentic slots are in that state.

The published rows

Benchmark Headline result Last re-run Full results
Cross-tool block rate · 210 tool calls 100% blocked (106/106, 95% CI 96.5–100%) · 0% false-positive (0/104, 95% CI 0–3.6%) · p50 ~3µs · 6 of 10 OWASP ASI slots measured; ASI07–ASI10 are n=0 2026-11-10 Cross-tool block rate results
sandbox=True dispatch · contract parity 4/4 annotated-contract probes refused on the sandbox path (0/4 before v0.10.6) · 204/204 verdicts agree with the local path, 100 policy items admitted and 100 refused on both. The 2026-09-16 run's 200/200 policy parity was vacuous: every policy item was refused on both paths 2026-11-10 Sandbox dispatch parity arm
Least-privilege · ToolPrivBench, 100 scenarios (subset) 100% over-privileged blocked · 100% low-privileged allowed · OPUR 100% → 0% 2026-11-10 ToolPrivBench results on GitHub
Indirect prompt injection · AgentDojo, all 4 suites 86.0% of injection→target tool-calls blocked (AgentDojo v1.2.1, whose CLI defaults to v1.2.2; 524/609 pairs that have a tool-call target, of 949 attacked pairs; deterministic bound). Model-in-the-loop ASR 45% → 10% (gpt-4o-mini, 2026-07-31), but that is one model family on a 60-pair subset, under AgentDojo's fixed-template attack, not an adaptive one. Replay of AgentDojo's published Claude 3.x runs (2026-10-12): the per-task policy refuses an emitted target call in 433/532 successful attacks (81%), and some call in 70/412 successful benign runs (17%). Claude 5.5 model-in-the-loop (168 pairs/arm, 2026-10-16) is a near-null: Haiku and Sonnet resist this fixed-template attack unaided (undefended ASR 5-8%), so airlock's realised reduction is +0-1% (pooled +0%, 336 attacked trajectories, Wilson CI [4%, 9%]) at a +14% benign-refusal cost; against models this robust the 86% deterministic bound is the informative measure 2026-11-10 AgentDojo results on GitHub
Tool poisoning · MCPTox, 1,348 cases Direct execution of the poisoned tool refused 100% under G (tool allowlist) and LP (1650/1650 API-model DE calls; G0 contract-only 2.4%), benign-proxy false-refusal 2.4% (G) / 4.1% (LP), over a free judge-free replay of 27 models' recorded calls. Fresh Claude 5.5 two-arm run (2026-10-16): G/LP refuse every direct poisoned-tool call (Haiku 74/74, Sonnet 42/42, Opus 26/26) at 0-6% benign cost; poisoned-prompt API-refusal rises with capability (Haiku 0%, Sonnet 12.7%, Opus 24.1%) 2026-11-10 MCPTox results
vs. native MCP gateway · 12 malformed payloads airlock 12/12 blocked · Docker MCP Gateway 0/12 · 0% false-positive on both 2026-11-03 Native MCP gateway head-to-head
Prompt injection across agent harnesses A null result, published as one. claude-code and codex each acted on the planted script 0/36, and ignored the benign twin too (claude-code 0/36, codex 0/5), so this is indifference to the channel, not detection. Weaker than the 2026-08-26 run: the one benign action that made codex's zero a choice did not reproduce. claude-code (model claude-opus-4-8) finished all 72 of its cells; codex ran 41 before its plan's usage limit stopped it; cursor-agent errored on every cell and is not measured (113 cells measured, 103 errored cells excluded from the denominator). The last complete matrix is 2026-10-03, 144/144 cells 2026-11-03 Multi-harness prompt injection
MCP spec conformance · @modelcontextprotocol/conformance Run against the wire-path validators, outcome published in full. Not a full MCP server/client conformance pass — airlock is a request validator, not a server, so it does not claim one 2026-11-10 MCP conformance results on GitHub

The Last re-run column is the same date the freshness gate reads out of README.md, and check_benchmark_freshness.py asserts the two agree. If a row here disagrees with the README, the build fails rather than publishing two different dates for one run.

Pages on this site

  • Cross-tool block rate — the flagship corpus: 210 tool calls, 106 malicious and 104 benign, with the per-OWASP-ASI breakdown and the sandbox=True dispatch arm.
  • Guard-suite corpus — the full per-class, false-positive-aware table over the versioned exploit corpus. A self-corpus, and labelled as one.
  • Multi-harness prompt injection — the matched-pair design, and why the benign control is the only reason the injected column means anything.
  • Native MCP gateway head-to-head — airlock against Docker's MCP Gateway on the same malformed payloads.
  • MCP gateway payload gap — what the gateway comparison does and does not establish.

Rows whose full results live in the benchmark packages rather than on this site link to GitHub in the table above.

Re-running them

Every row is reproducible from the repository. The deterministic ones need nothing but a checkout:

python -m benchmarks.blockrate          # block rate, latency, sandbox dispatch arm
python -m benchmarks.toolprivbench      # least-privilege
make benchmark                          # regenerate BENCHMARK.md from the corpus

Three are not free. benchmarks.agentdojo.run --model spends API credit, benchmarks.harness_injection spends the agent CLIs' plan quota, and benchmarks.vs_gateway needs a Docker daemon. The multi-harness run is roughly 100 minutes of paid API time for a full --trials 18 matrix, which is why it takes a --checkpoint flag and refuses to do anything without an explicit --run:

python -m benchmarks.harness_injection --trials 18 --write --checkpoint ckpt.json --run

python -m benchmarks.harness_injection on its own is a dry run. It prints the cells it would execute and the statistical power that many trials would buy, so the cost is known before it is spent.