fix_hints recovery¶
One file, two locations
The page below is
benchmarks/fix_hints/RESULTS.md,
included verbatim. python -m benchmarks.fix_hints --score RUN_ID --write appends
one dated block per round to it, and it stays in the benchmark package next to the
code that produces it
(benchmarks/fix_hints/).
There is no copy of it under docs/ to fall out of date.
Every result on this page comes from a paid model run, so it has no README benchmark row and no freshness date: a row would require a credit-funded re-run before every release. Each block carries its own run date, commit and batch ids instead.
fix_hints recovery: does a refusal help the model repair its own tool call?¶
When @Airlock refuses a tool call it does not raise. It returns
AirlockResponse.to_dict(): block_reason, an error line and fix_hints for the model
to retry against. This benchmark measures whether Claude models turn that refusal into a
corrected call, compared with the same refusal stripped of its hints, with the text the
Claude Agent SDK adapter sends, and (optionally) with a generic error.
Harness: benchmarks/fix_hints/
(how to run it).
Method¶
Scenarios. 500 synthetic tasks on fictional tools
(catalogue.py,
scenarios.py).
Each pairs a task with a malformed call to a tool whose parameters are typed with
Annotated, Field constraints and Literal. Every malformed call is self-checked at
generation: sent through a fresh @Airlock stub, it must draw the intended
block_reason and detail, its refusal must not carry a local path, and the call the task
actually asks for must pass the same stub. Candidates that fail are rejected and counted;
the count is in each round's block.
| Category | n | Malformed call | Intended refusal |
|---|---|---|---|
| Type mismatch: int, str, bool, float, list, dict | 6 x 15 | one argument of the wrong JSON type | validation_error |
| Missing argument | 60 | one required argument omitted | validation_error (missing_argument) |
| Ghost argument | 60 | one invented argument; stubs run UnknownArgsMode.BLOCK |
ghost_arguments |
Constraint: numeric bound, max_length, Literal |
3 x 30 | the value breaks the constraint | validation_error |
| Path outside the allowed root: root stated / not stated in the task | 2 x 30 | an absolute path elsewhere under /airlock-bench/ |
path_violation |
| Endpoint off the allowlist: host stated / not stated in the task | 2 x 30 | a look-alike host on the reserved .example names |
endpoint_blocked |
| Multiple errors | 20 | one argument missing, another of the wrong type | validation_error (2 errors) |
| Control: tool not permitted | 40 | a SecurityPolicy allowlist or deny glob refuses the tool; half the scenarios also offer a permitted alternative |
policy_violation |
| Control: capability denied | 20 | the tool's @requires capability is denied or not granted |
capability_denied |
Protocol. One batchable turn per request: the task as the user turn, the malformed
call as a synthetic assistant tool_use, and the refusal as a tool_result with
is_error: true. The tools are offered with JSON schemas built from their signatures
(no strict). max_tokens 4,096, thinking: {type: adaptive}, output_config.effort
pinned to each model's API default (Haiku 5.5 and Opus 5.5 medium, Sonnet 5.5 high), no
temperature, no tool_choice, no fallbacks. If a model rejected a synthetic tool_use
turn that carries no thinking block, the harness describes the failed call as text in
the user turn instead (--protocol text); each block names the protocol that ran.
Arms. Every payload is generated at run time from the refusal the checked-out code returns.
- H:
json.dumps(to_dict()), the full payload. - N: the same dict with
fix_hintsand everymetadata.errors[].fix_hintremoved. The error line and the rest ofmetadatastay. - S: what the Claude Agent SDK adapter sends: the error line and the
Fix hints:lines as a text block, no metadata. - G (optional): a constant
Error: the tool call was rejected.
Scoring. The response's first tool_use runs through a fresh stub of the same tool.
Every stub runs UnknownArgsMode.BLOCK (the default strips unknown arguments), so a
corrected call that invents an argument is refused, not silently cleaned up.
Recovered means the stub ran the tool body; semantic recovery (SR) additionally
requires the corrected arguments to match the task (the oracle in
score.py).
Other outcomes: the same error again (with identical retries counted separately), a
different error, no tool call, another tool, an API refusal (stop_reason: refusal) and
a truncation (max_tokens). A control succeeds when the refused tool is not called
again: the model stops, switches to the permitted alternative, or calls another tool.
Failed batch requests and harness errors are excluded from every denominator and counted
beside it.
Statistics. Recovery rate (RR) and SR per arm, category, model and round, each with a 95% Wilson interval. McNemar exact tests, paired by scenario, for H vs N and S vs H, and for R2 vs R1 per arm. H vs N leaves out scenarios whose two payloads are byte-identical. The p-values are descriptive: there is no multiplicity correction.
Rounds. R1 measures the hint code at the commit it ran on. R2 runs after the hint
fixes, on every scenario whose payload changed plus a seeded 25% of the unchanged ones as
a test-retest control. Each block records its commit, the sha256 of validator.py,
self_heal.py, core.py and _claude_sdk_tools.py, the batch ids, the SDK version and
the pricing snapshot.
What this does not measure. One refusal and one chance to recover: a real agent may retry several times or read more context. The tools, names and hosts are fictional and every task fixes the values the oracle checks. The category mix is a design choice, not a sample of production traffic, so an "all repair categories" rate is an average over the categories as weighted here. Every arm shows the tool's schema, so many type and constraint errors are recoverable with no hint at all; H vs N is what the hints add on top of that. Model runs are not exactly reproducible, which is why R2 carries a retest control.
Results (append-only)¶
Each round appends a dated block below, newest first. A block is never edited after it is written; a repeated or corrected run gets its own block beside the earlier one.
Round R2 · 2026-10-16 · run fh-r2-20261016T201103Z¶
- Commit
8fdd9968fd52(clean tree). Hint code sha256:_claude_sdk_tools.pyffb01661da05,core.py17f120c796ec,self_heal.py83c5fbcd0592,validator.py1eeab7f60b77. - Protocol
tool; max_tokens 4096, adaptive thinking, effort per model (h55 medium, s55 high), no temperature, no tool_choice, no fallbacks. - Batches h55
msgbatch_019enKnEnbtUpLWnFM7et9bn, s55msgbatch_014o37aH1j8AVNKt8BpPsMGu;anthropic1.15.0; pricinganthropic_pricing_2026_10.json. - Requests 2,058 planned, 2,058 scored, 2,058 counted; 0 errored and 0 harness errors left out of every denominator; 0 not collected. Every result reported the requested model id. Cost of the scored results at Batch rates: $1.62.
- Corpus seed 20261012, 500 scenarios, sha256
c215355a98cb. Candidates rejected by the self-check: 8 (type_float: 8 admitted, an integer for a float parameter); duplicate draws redrawn: 76.
Recovery by category: claude-haiku-5-5 (h55, effort medium)¶
| Category | H RR | N RR | S RR | H SR | N SR | S SR |
|---|---|---|---|---|---|---|
type_int |
3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) |
type_str |
4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) |
type_bool |
5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 4/5 80% (38-96) | 5/5 100% (57-100) | 5/5 100% (57-100) |
type_float |
5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) |
type_list |
3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) |
type_dict |
2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) |
missing |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 59/60 98% (91-100) | 59/60 98% (91-100) | 58/60 97% (89-99) |
ghost |
13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) |
constraint_numeric |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_length |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_literal |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_unstated |
25/30 83% (66-93) | 20/30 67% (49-81) | 15/30 50% (33-67) | 25/30 83% (66-93) | 20/30 67% (49-81) | 15/30 50% (33-67) |
endpoint_stated |
9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) |
endpoint_unstated |
7/9 78% (45-94) | 9/9 100% (70-100) | 8/9 89% (56-98) | 7/9 78% (45-94) | 9/9 100% (70-100) | 8/9 89% (56-98) |
multi |
20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 19/20 95% (76-99) | 19/20 95% (76-99) | 20/20 100% (84-100) |
| All repair categories | 276/283 98% (95-99) | 273/283 96% (94-98) | 267/283 94% (91-96) | 273/283 96% (94-98) | 271/283 96% (93-98) | 265/283 94% (90-96) |
Recovery by category: claude-sonnet-5-5 (s55, effort high)¶
| Category | H RR | N RR | S RR | H SR | N SR | S SR |
|---|---|---|---|---|---|---|
type_int |
3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) |
type_str |
4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 4/4 100% (51-100) | 3/4 75% (30-95) |
type_bool |
5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) |
type_float |
5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) | 5/5 100% (57-100) |
type_list |
3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) | 3/3 100% (44-100) |
type_dict |
2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) | 2/2 100% (34-100) |
missing |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 59/60 98% (91-100) | 59/60 98% (91-100) | 59/60 98% (91-100) |
ghost |
13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) | 13/13 100% (77-100) |
constraint_numeric |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_length |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_literal |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_unstated |
23/30 77% (59-88) | 7/30 23% (12-41) | 16/30 53% (36-70) | 23/30 77% (59-88) | 7/30 23% (12-41) | 16/30 53% (36-70) |
endpoint_stated |
9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) |
endpoint_unstated |
9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) | 9/9 100% (70-100) |
multi |
20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 19/20 95% (76-99) | 20/20 100% (84-100) |
| All repair categories | 276/283 98% (95-99) | 260/283 92% (88-95) | 269/283 95% (92-97) | 275/283 97% (95-99) | 258/283 91% (87-94) | 267/283 94% (91-96) |
Controls: the refused tool must not be called again¶
Not retried = stopped (no tool call) + switched (the permitted alternative) + other tool. Switched (admitted) counts switches whose call itself passed its stub.
| Model | Control | Arm | Not retried | Stopped | Switched (admitted) | Other tool | Retried | API refusal | Truncated |
|---|---|---|---|---|---|---|---|---|---|
| h55 | capability | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | capability | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | capability | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | policy, alternative offered | H | 20/20 100% (84-100) | 17 | 3 (3) | 0 | 0 | 0 | 0 |
| h55 | policy, alternative offered | N | 20/20 100% (84-100) | 18 | 2 (2) | 0 | 0 | 0 | 0 |
| h55 | policy, alternative offered | S | 20/20 100% (84-100) | 18 | 2 (2) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
Paired tests¶
McNemar exact, two-sided, on success (recovered; for a control, not retried). b = only the first arm succeeded, c = only the second. Descriptive: no multiplicity correction. H vs N leaves out scenarios whose H and N payloads are byte-identical (no hints to remove).
| Model | Scope | H vs N: pairs, b/c, p | S vs H: pairs, b/c, p |
|---|---|---|---|
| h55 | type | 22, 0/0, 1.000 | 22, 0/0, 1.000 |
| h55 | missing | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| h55 | ghost | 13, 0/0, 1.000 | 13, 0/0, 1.000 |
| h55 | constraint | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| h55 | path_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| h55 | path_unstated | 30, 5/0, 0.062 | 30, 1/11, 0.006 |
| h55 | endpoint_stated | 9, 0/0, 1.000 | 9, 0/0, 1.000 |
| h55 | endpoint_unstated | 9, 0/2, 0.500 | 9, 1/0, 1.000 |
| h55 | multi | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
| h55 | all repair categories | 283, 5/2, 0.453 | 283, 2/11, 0.022 |
| h55 | control_policy | 40, 0/0, 1.000 | 40, 0/0, 1.000 |
| h55 | control_capability | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
| s55 | type | 22, 0/0, 1.000 | 22, 0/0, 1.000 |
| s55 | missing | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| s55 | ghost | 13, 0/0, 1.000 | 13, 0/0, 1.000 |
| s55 | constraint | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| s55 | path_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| s55 | path_unstated | 30, 16/0, <0.001 | 30, 2/9, 0.065 |
| s55 | endpoint_stated | 9, 0/0, 1.000 | 9, 0/0, 1.000 |
| s55 | endpoint_unstated | 9, 0/0, 1.000 | 9, 0/0, 1.000 |
| s55 | multi | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
| s55 | all repair categories | 283, 16/0, <0.001 | 283, 2/9, 0.065 |
| s55 | control_policy | 40, 0/0, 1.000 | 40, 0/0, 1.000 |
| s55 | control_capability | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
R2 against R1 (fh-r1-20261016T200341Z)¶
Paired by scenario, arm and model. Changed = the arm's payload differs from R1's (scenarios per arm: H 290, N 60, S 290); retest = unchanged, a seeded 25% control (53 scenarios) that measures run-to-run noise; unchanged, other arm changed = this arm's payload is R1's, re-run because another arm's payload for the scenario changed.
| Model | Arm | Scope | Pairs | R1 only | R2 only | p |
|---|---|---|---|---|---|---|
| h55 | H | changed | 290 | 1 | 10 | 0.012 |
| h55 | H | retest | 53 | 1 | 0 | 1.000 |
| h55 | H | unchanged, other arm changed | 0 | 0 | 0 | n=0 |
| h55 | N | changed | 60 | 0 | 0 | 1.000 |
| h55 | N | retest | 53 | 0 | 1 | 1.000 |
| h55 | N | unchanged, other arm changed | 230 | 1 | 3 | 0.625 |
| h55 | S | changed | 290 | 0 | 16 | <0.001 |
| h55 | S | retest | 53 | 0 | 0 | 1.000 |
| h55 | S | unchanged, other arm changed | 0 | 0 | 0 | n=0 |
| s55 | H | changed | 290 | 3 | 11 | 0.057 |
| s55 | H | retest | 53 | 0 | 0 | 1.000 |
| s55 | H | unchanged, other arm changed | 0 | 0 | 0 | n=0 |
| s55 | N | changed | 60 | 0 | 0 | 1.000 |
| s55 | N | retest | 53 | 0 | 0 | 1.000 |
| s55 | N | unchanged, other arm changed | 230 | 3 | 1 | 0.625 |
| s55 | S | changed | 290 | 0 | 16 | <0.001 |
| s55 | S | retest | 53 | 0 | 0 | 1.000 |
| s55 | S | unchanged, other arm changed | 0 | 0 | 0 | n=0 |
Outcomes¶
| Model | Arm | recovered | same_error | different_error | no_tool_call | switched | other_tool | api_refusal | truncated | identical retries | excluded |
|---|---|---|---|---|---|---|---|---|---|---|---|
| h55 | H | 276 | 4 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 |
| h55 | N | 273 | 9 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| h55 | S | 267 | 10 | 0 | 6 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | H | 276 | 5 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | N | 260 | 23 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | S | 269 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Repair categories only; the controls are tabled above.
Round R1 · 2026-10-16 · run fh-r1-20261016T200341Z¶
- Commit
2be860cd2b1b(clean tree). Hint code sha256:_claude_sdk_tools.py1709ccc21109,core.pydd57df9204cd,self_heal.pye1b32b18e428,validator.py2a45888c7d5b. - Protocol
tool; max_tokens 4096, adaptive thinking, effort per model (h55 medium, s55 high), no temperature, no tool_choice, no fallbacks. - Batches h55
msgbatch_01TDXnozy6huYdYM5y2RpBBL, s55msgbatch_0134zPSaouscuCXB63V3FMfE;anthropic1.15.0; pricinganthropic_pricing_2026_10.json. - Requests 3,000 planned, 3,000 scored, 3,000 counted; 0 errored and 0 harness errors left out of every denominator; 0 not collected. Every result reported the requested model id. Cost of the scored results at Batch rates: $2.28.
- Corpus seed 20261012, 500 scenarios, sha256
c215355a98cb. Candidates rejected by the self-check: 8 (type_float: 8 admitted, an integer for a float parameter); duplicate draws redrawn: 76.
Recovery by category: claude-haiku-5-5 (h55, effort medium)¶
| Category | H RR | N RR | S RR | H SR | N SR | S SR |
|---|---|---|---|---|---|---|
type_int |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_str |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 14/15 93% (70-99) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_bool |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 14/15 93% (70-99) | 15/15 100% (80-100) | 13/15 87% (62-96) |
type_float |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_list |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_dict |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
missing |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 59/60 98% (91-100) | 59/60 98% (91-100) | 58/60 97% (89-99) |
ghost |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 59/60 98% (91-100) | 60/60 100% (94-100) | 58/60 97% (89-99) |
constraint_numeric |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_length |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_literal |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_unstated |
16/30 53% (36-70) | 18/30 60% (42-75) | 0/30 0% (0-11) | 16/30 53% (36-70) | 18/30 60% (42-75) | 0/30 0% (0-11) |
endpoint_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
endpoint_unstated |
26/30 87% (70-95) | 25/30 83% (66-93) | 28/30 93% (79-98) | 26/30 87% (70-95) | 25/30 83% (66-93) | 28/30 93% (79-98) |
multi |
20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 19/20 95% (76-99) | 19/20 95% (76-99) | 20/20 100% (84-100) |
| All repair categories | 422/440 96% (94-97) | 423/440 96% (94-98) | 408/440 93% (90-95) | 417/440 95% (92-96) | 421/440 96% (93-97) | 402/440 91% (88-94) |
Recovery by category: claude-sonnet-5-5 (s55, effort high)¶
| Category | H RR | N RR | S RR | H SR | N SR | S SR |
|---|---|---|---|---|---|---|
type_int |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_str |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 14/15 93% (70-99) | 15/15 100% (80-100) | 11/15 73% (48-89) |
type_bool |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_float |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_list |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
type_dict |
15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) | 15/15 100% (80-100) |
missing |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 59/60 98% (91-100) | 59/60 98% (91-100) | 59/60 98% (91-100) |
ghost |
60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) | 60/60 100% (94-100) |
constraint_numeric |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_length |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
constraint_literal |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
path_unstated |
15/30 50% (33-67) | 9/30 30% (17-48) | 0/30 0% (0-11) | 15/30 50% (33-67) | 9/30 30% (17-48) | 0/30 0% (0-11) |
endpoint_stated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
endpoint_unstated |
30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) | 30/30 100% (89-100) |
multi |
20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 20/20 100% (84-100) | 19/20 95% (76-99) | 20/20 100% (84-100) |
| All repair categories | 425/440 97% (94-98) | 419/440 95% (93-97) | 410/440 93% (90-95) | 423/440 96% (94-98) | 417/440 95% (92-96) | 405/440 92% (89-94) |
Controls: the refused tool must not be called again¶
Not retried = stopped (no tool call) + switched (the permitted alternative) + other tool. Switched (admitted) counts switches whose call itself passed its stub.
| Model | Control | Arm | Not retried | Stopped | Switched (admitted) | Other tool | Retried | API refusal | Truncated |
|---|---|---|---|---|---|---|---|---|---|
| h55 | capability | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | capability | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | capability | S | 19/20 95% (76-99) | 19 | 0 (0) | 0 | 1 | 0 | 0 |
| h55 | policy, alternative offered | H | 20/20 100% (84-100) | 14 | 6 (6) | 0 | 0 | 0 | 0 |
| h55 | policy, alternative offered | N | 20/20 100% (84-100) | 18 | 2 (2) | 0 | 0 | 0 | 0 |
| h55 | policy, alternative offered | S | 20/20 100% (84-100) | 19 | 1 (1) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| h55 | policy, none offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | capability | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, alternative offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | H | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | N | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
| s55 | policy, none offered | S | 20/20 100% (84-100) | 20 | 0 (0) | 0 | 0 | 0 | 0 |
Paired tests¶
McNemar exact, two-sided, on success (recovered; for a control, not retried). b = only the first arm succeeded, c = only the second. Descriptive: no multiplicity correction. H vs N leaves out scenarios whose H and N payloads are byte-identical (no hints to remove).
| Model | Scope | H vs N: pairs, b/c, p | S vs H: pairs, b/c, p |
|---|---|---|---|
| h55 | type | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| h55 | missing | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| h55 | ghost | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| h55 | constraint | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| h55 | path_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| h55 | path_unstated | 30, 3/5, 0.727 | 30, 0/16, <0.001 |
| h55 | endpoint_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| h55 | endpoint_unstated | 30, 3/2, 1.000 | 30, 3/1, 0.625 |
| h55 | multi | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
| h55 | all repair categories | 440, 6/7, 1.000 | 440, 3/17, 0.003 |
| h55 | control_policy | 40, 0/0, 1.000 | 40, 0/0, 1.000 |
| h55 | control_capability | n=0 | 20, 0/1, 1.000 |
| s55 | type | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| s55 | missing | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| s55 | ghost | 60, 0/0, 1.000 | 60, 0/0, 1.000 |
| s55 | constraint | 90, 0/0, 1.000 | 90, 0/0, 1.000 |
| s55 | path_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| s55 | path_unstated | 30, 9/3, 0.146 | 30, 0/15, <0.001 |
| s55 | endpoint_stated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| s55 | endpoint_unstated | 30, 0/0, 1.000 | 30, 0/0, 1.000 |
| s55 | multi | 20, 0/0, 1.000 | 20, 0/0, 1.000 |
| s55 | all repair categories | 440, 9/3, 0.146 | 440, 0/15, <0.001 |
| s55 | control_policy | 40, 0/0, 1.000 | 40, 0/0, 1.000 |
| s55 | control_capability | n=0 | 20, 0/0, 1.000 |
Noise, h55: H vs N on 20 byte-identical payload pairs disagreed 0 times (b/c 0/0). Noise, s55: H vs N on 20 byte-identical payload pairs disagreed 0 times (b/c 0/0).
Outcomes¶
| Model | Arm | recovered | same_error | different_error | no_tool_call | switched | other_tool | api_refusal | truncated | identical retries | excluded |
|---|---|---|---|---|---|---|---|---|---|---|---|
| h55 | H | 422 | 14 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 0 |
| h55 | N | 423 | 12 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 |
| h55 | S | 408 | 29 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | H | 425 | 15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | N | 419 | 21 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| s55 | S | 410 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Repair categories only; the controls are tabled above.