Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Not on the map. Why?
Abstract (arXiv)
LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 15 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
11 of 15 references.
References · 11
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
- 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
- 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
- 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
- 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
- 2026Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biasescited by 1
Built on by
Not cited yet.