mlsanityData to 7 Oct 2026

Tuning LLM Judge Design Decisions for 1/1000 of the Cost

arXiv 2501.17178 ↗First posted 24 Jan 2025cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention. While several approaches have been proposed, many confounding factors are present between different papers. For instance the model, the prompt and other hyperparameters are typically changed at the same time making apple-to-apple comparisons challenging. In this paper, we propose to systematically analyze and tune the hyperparameters of LLM judges. To alleviate the high cost of evaluating a judge, we propose to leverage multi-objective multi-fidelity which allows to find judges that trade accuracy for cost and also significantly reduce the cost of the search. Our method identifies judges that not only outperform existing benchmarks in accuracy and cost-efficiency but also utilize open-weight models, ensuring greater accessibility and reproducibility. The code to reproduce our experiments is available at this repository https://github.com/geoalgo/judgetuning .

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

21 of 28 references.

References · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  10. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  12. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
9 more
  1. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
  2. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  3. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 38
  4. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 35
  5. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  6. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  7. 2021Multi-objective Asynchronous Successive Halvingcited by 22
  8. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  9. 2021A multi-objective perspective on jointly tuning hardware and hyperparameterscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputscited by 7
  2. 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
  3. 2025Single- vs. Dual-Prompt Dialogue Generation with LLMs for Job Interviews in Human Resourcescited by 2
  4. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2
  5. 2026LLM-MC-Affect: LLM-Based Monte Carlo Modeling of Affective Trajectories and Latent Ambiguity for Interpersonal Dynamic Insightcited by 1
  6. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  7. 2026Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learningcited by 1
  8. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  9. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0
  10. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  11. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0