# How to Correctly Report LLM-as-a-Judge Evaluations

arXiv: https://arxiv.org/abs/2511.21140 · HTML: https://mlsanity.com/paper/2511.21140/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4](https://mlsanity.com/paper/2403.02839/) (2024)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics](https://mlsanity.com/paper/2501.14883/) (2025)
- [Improving the output quality of official statistics based on machine learning algorithms](https://mlsanity.com/paper/2103.00834/) (2021)
- [Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality](https://mlsanity.com/paper/2510.18982/) (2025)

## Built on by (most cited first, 30 of 30)

- [PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization](https://mlsanity.com/paper/2603.26535/) (2026)
- [Quantifying and Mitigating Self-Preference Bias of LLM Judges](https://mlsanity.com/paper/2604.22891/) (2026)
- [Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines](https://mlsanity.com/paper/2606.15474/) (2026)
- [Efficient Inference for Noisy LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2601.05420/) (2026)
- [AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs](https://mlsanity.com/paper/2604.22937/) (2026)
- [Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why](https://mlsanity.com/paper/2606.00093/) (2026)
- [Bias and Uncertainty in LLM-as-a-Judge Estimation](https://mlsanity.com/paper/2605.06939/) (2026)
- [REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems](https://mlsanity.com/paper/2608.10669/) (2026)
- [MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attribution](https://mlsanity.com/paper/2512.05958/) (2025)
- [Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems](https://mlsanity.com/paper/2512.11150/) (2025)
- [ReEfBench: Quantifying the Reasoning Efficiency of LLMs](https://mlsanity.com/paper/2601.03550/) (2026)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models](https://mlsanity.com/paper/2601.22336/) (2026)
- [Open-Ended Task Discovery via Bayesian Optimization](https://mlsanity.com/paper/2605.07572/) (2026)
- [Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges](https://mlsanity.com/paper/2603.17172/) (2026)
- [Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents](https://mlsanity.com/paper/2606.08200/) (2026)
- [SysAdmin: Measuring Instrumental Power-Seeking in Frontier AI](https://mlsanity.com/paper/2607.18239/) (2026)
- [Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation](https://mlsanity.com/paper/2608.26638/) (2026)
- [SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews](https://mlsanity.com/paper/2609.05505/) (2026)
- [ICDAR2026 Competition on Multimodal Reasoning over Documents in Multiple Domains](https://mlsanity.com/paper/2609.25055/) (2026)
- [BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluation](https://mlsanity.com/paper/2601.18253/) (2026)
- [Task-Aware Calibration: Provably Optimal Decoding in LLMs](https://mlsanity.com/paper/2605.10202/) (2026)
- [ConVAWG: A Retrieval-Grounded Framework for Controlled Synthetic Dialogue Generation in Violence Against Women and Girls](https://mlsanity.com/paper/2608.11200/) (2026)
- [Human-Anchored Factuality Evaluation with Strategic Annotation](https://mlsanity.com/paper/2609.00494/) (2026)
- [DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration](https://mlsanity.com/paper/2609.31215/) (2026)
- [LLM Judge Validation Under Sparse Overlap: From Inference to Design](https://mlsanity.com/paper/2609.31857/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)
- [Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions](https://mlsanity.com/paper/2610.09229/) (2026)
- [Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment](https://mlsanity.com/paper/2610.15961/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches.