mlsanityData to 14 Oct 2026

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

arXiv 2603.12520 ↗First posted 12 Mar 2026cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

31 of 50 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
19 more
  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  7. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  8. 2018Categorizing Variants of Goodhart's Lawcited by 149
  9. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  10. 2024How to Evaluate Reward Models for RLHFcited by 88
  11. 2024Bayesian Calibration of Win Rate Estimation with LLM Evaluatorscited by 16
  12. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  13. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  14. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  15. 2024The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluatorscited by 11
  16. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 11
  17. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 10
  18. 2026Judge Reliability Harness: Stress Testing the Reliability of LLM Judgescited by 8
  19. 2025Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognitioncited by 3
  2. 2026Permutation-Consensus Listwise Judging for Robust Factuality Evaluationcited by 1
  3. 2026Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Controlcited by 1
  4. 2026Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gainscited by 1
  5. 2026Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearchcited by 0
  6. 2026Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guaranteescited by 0
  7. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0
  8. 2026Three Ways Classical Test Theory Can Mislead About LLM Judgescited by 0
  9. 2026Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Modelscited by 0