mlsanityData to 7 Oct 2026

LLM-as-Judge on a Budget

arXiv 2602.15481 ↗First posted 17 Feb 2026cs.LGcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pairs. Since LLM judgments are stochastic, practitioners commonly query each pair multiple times to estimate mean scores accurately. This raises a critical challenge: given a fixed computational budget B, how to optimally allocate queries across K prompt-response pairs to minimize estimation error? We present a principled variance-adaptive approach leveraging multi-armed bandit theory and concentration inequalities. Our method dynamically allocates queries based on estimated score variances, concentrating resources where uncertainty is highest. Further, our algorithm is shown to achieve a worst-case score-estimation error of tildeOleft(√fracsumi=1K σi2Bright), σi2 being the unknown score variance for pair i ∈ [K] with near-optimal budget allocation. Experiments on Summarize-From-Feedback and HelpSteer2 demonstrate that our method significantly outperforms uniform allocation, reducing worst-case estimation error while maintaining identical budgets. Our work establishes a theoretical foundation for efficient LLM evaluation with practical implications for AI safety, model alignment, and automated assessment at scale.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 29 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

19 of 29 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
7 more
  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  2. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  4. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  5. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  6. 2023LLMEval: A Preliminary Study on How to Evaluate Large Language Modelscited by 23
  7. 2023Fixed-Budget Best-Arm Identification with Heterogeneous Reward Variancescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluationcited by 4
  2. 2026Dynamically Allocating Evaluation Effort for Model Rankingcited by 0
  3. 2026Speculative Evaluation of Stochastic LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniques
  2. 2025Evaluating Scoring Bias in LLM-as-a-Judge
  3. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols
  4. 2025Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
  5. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  6. 2025Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
  7. 2025Improve LLM-as-a-Judge Ability as a General Ability
  8. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization