mlsanityData to 7 Oct 2026

Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

arXiv 2407.18370 ↗First posted 25 Jul 2024cs.LG cs.CLcited by 99

Not on the map. Why?

Abstract (arXiv)

We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework, human agreement can be provably guaranteed -- such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce Simulated Annotators, a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Evaluation, where we use cheaper models as initial judges and escalate to stronger models only when necessary -- again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guarantees over 80% human agreement with almost 80% test coverage.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

30 of 37 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2017Selective Classification for Deep Neural Networkscited by 799
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
18 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  6. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  8. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  9. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  10. 2023Fine-tuning Language Models for Factualitycited by 219
  11. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  12. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  13. 2024Language Models with Conformal Factuality Guaranteescited by 128
  14. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  15. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  16. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  17. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  18. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 28; 2026: 65.

99 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  6. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  7. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  8. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  9. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  10. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 25
  11. 2024A Survey on the Honesty of Large Language Modelscited by 21
  12. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
38 more
  1. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
  2. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  3. 2025Cost-Optimal Active AI Model Evaluationcited by 14
  4. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  5. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  6. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  7. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  8. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validitycited by 11
  9. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 10
  10. 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
  11. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applicationscited by 7
  12. 2025CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answeringcited by 6
  13. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  14. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  15. 2025Auto-Prompt Ensemble for LLM Judgecited by 5
  16. 2026SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibrationcited by 5
  17. 2026JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Placescited by 5
  18. 2025Continual Learning, Not Training: Online Adaptation For Agentscited by 4
  19. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  20. 2026LLM Performance Predictors: Learning When to Escalate in Hybrid Human-AI Moderation Systemscited by 4
  21. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 4
  22. 2026When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generationcited by 4
  23. 2025THiNK: Can Large Language Models Think-aloud?cited by 3
  24. 2025LLMs Judging LLMs: A Simplex Perspectivecited by 3
  25. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  26. 2026Invisible failures in human-AI interactionscited by 3
  27. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  28. 2026Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascadescited by 3
  29. 2026Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?cited by 3
  30. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 3
  31. 2026Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Whycited by 3
  32. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agentscited by 3
  33. 2024The Superalignment of Superhuman Intelligence with Large Language Modelscited by 2
  34. 2025Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluationcited by 2
  35. 2025Becoming Experienced Judges: Selective Test-Time Learning for Evaluatorscited by 2
  36. 2025Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systemscited by 2
  37. 2026Small Reward Models via Backward Inferencecited by 2
  38. 2026BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generationcited by 2

50 most cited of 99 All 99 (JSON)