mlsanityData to 14 Oct 2026

Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes

arXiv 2606.07834 ↗First posted 5 Jun 2026cs.SE cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Evidence-grounded fact verification increasingly uses LLM judges to turn evidence into terminal verdicts. Many task contracts deliberately include non-directional verdicts - Conflicting for materially mixed evidence, Not Enough Evidence for absent evidence - so that a system can decline to assert a direction. Returning Supports or Refutes on a claim whose authorized verdict is non-directional is directional overcommitment: a label error whose output is a direction the contract did not authorize. Its conflict instance - both strands present, one of them returned - is Cherry-pick Override (CPO); unlike the insufficiency instance it cannot be repaired by retrieving more, because nothing was missing. On AVeriTeC's gold-Conflicting subset (N_C=150) a four-option typed panel commits directionally on 18.7% of claims, and four frozen contemporary judges (two proprietary, two open-weight) on 21.3-48.0%, with 24.0-38.0% on VitaminC-Mixed, a constructed stress test on which the ranking reverses. These are conservative lower bounds: the shared prompt specifies Conflicting in detail, forbids collapsing to a direction, and demonstrates the label three times. The failure is not specific to conflict: overcommitment on gold-Insufficient cases runs at a comparable rate for every judge and exceeds CPO for three of four on VitaminC-Mixed. Asked in a fresh context, two judges identify the conflict on about two thirds of their own CPO claims, against under 0.12 where they committed correctly: the recognition is available and does not reach the terminal action. Typed vocabulary, panel voting, confidence thresholds, a structural validator and self-decomposition each leave a substantial residual, and the last buys part of its reduction by relabelling one-sided evidence as conflicting. Every rate is dataset-defined; we report no human-confirmed prevalence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

12 of 21 references.

References · 12

  1. 2017Selective Classification for Deep Neural Networkscited by 852
  2. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  3. 2020Selective Question Answering under Domain Shiftcited by 238
  4. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  5. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  6. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  7. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  8. 2024The Automated Verification of Textual Claims (AVeriTeC) Shared Taskcited by 29
  9. 2024ECon: On the Detection and Resolution of Evidence Conflictscited by 18
  10. 2025Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answeringcited by 5
  11. 2025Explaining Sources of Uncertainty in Automated Fact-Checkingcited by 1
  12. 2026When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answeringcited by 1

Built on by

Not cited yet.