mlsanityData to 7 Oct 2026

How to Correctly Report LLM-as-a-Judge Evaluations

arXiv 2511.21140 ↗First posted 26 Nov 2025cs.LG cs.CL stat.APcited by 25

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 20 of its 44 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

20 of 44 references.

References · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  10. 2023Prediction-Powered Inferencecited by 214
  11. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  12. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
8 more
  1. 2023PPI++: Efficient Prediction-Powered Inferencecited by 103
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  3. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  4. 2023Cross-Prediction-Powered Inferencecited by 57
  5. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
  6. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metricscited by 8
  7. 2021Improving the output quality of official statistics based on machine learning algorithmscited by 5
  8. 2025Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimalitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 23.

25 papers.

Sort:

By area:

  1. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 9
  2. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 8
  3. 2026Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelinescited by 8
  4. 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluationcited by 7
  5. 2026AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputscited by 7
  6. 2026Bias and Uncertainty in LLM-as-a-Judge Estimationcited by 4
  7. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systemscited by 4
  8. 2025MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attributioncited by 3
  9. 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMscited by 3
  10. 2026Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Whycited by 3
  11. 2025Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systemscited by 2
  12. 2026Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Modelscited by 1
13 more
  1. 2026Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judgescited by 1
  2. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  3. 2026SysAdmin: Measuring Instrumental Power-Seeking in Frontier AIcited by 1
  4. 2026Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluationcited by 1
  5. 2026SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviewscited by 1
  6. 2026ICDAR2026 Competition on Multimodal Reasoning over Documents in Multiple Domainscited by 1
  7. 2026BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluationcited by 0
  8. 2026Task-Aware Calibration: Provably Optimal Decoding in LLMscited by 0
  9. 2026ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girlscited by 0
  10. 2026Human-Anchored Factuality Evaluation with Strategic Annotationcited by 0
  11. 2026DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibrationcited by 0
  12. 2026LLM Judge Validation Under Sparse Overlap: From Inference to Designcited by 0
  13. 2026How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstatscited by 0