Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
Not on the map. Why?
Abstract (arXiv)
Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
15 of 22 references.
References · 15
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
- 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
- 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
- 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMscited by 29
- 2026JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systemscited by 10
- 2026LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogycited by 7
- 2026The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steeringcited by 5
Built on by
Not cited yet.