mlsanityData to 14 Oct 2026

JuStRank: Benchmarking LLM Judges for System Ranking

arXiv 2412.09569 ↗First posted 12 Dec 2024cs.CL cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Given the rapid progress of generative AI, there is a pressing need to systematically compare and choose between the numerous models and configurations available. The scale and versatility of such evaluations make the use of LLM-based judges a compelling solution for this challenge. Crucially, this approach requires first to validate the quality of the LLM judge itself. Previous work has focused on instance-based assessment of LLM judges, where a judge is evaluated over a set of responses, or response pairs, while being agnostic to their source systems. We argue that this setting overlooks critical factors affecting system-level ranking, such as a judge's positive or negative bias towards certain systems. To address this gap, we conduct the first large-scale study of LLM judges as system rankers. System scores are generated by aggregating judgment scores over multiple system outputs, and the judge's quality is assessed by comparing the resulting system ranking to a human-based ranking. Beyond overall judge assessment, our analysis provides a fine-grained characterization of judge behavior, including their decisiveness and bias.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

32 of 49 references.

References · 32

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  11. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  12. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
20 more
  1. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  2. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  7. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  8. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  9. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  10. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  11. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  12. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  13. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 38
  14. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  15. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  16. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  17. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
  18. 2024Label-Efficient Model Selection for Text Generationcited by 19
  19. 2024Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluationcited by 2
  20. 2024A Measure of the System Dependence of Automated Metricscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 19.

29 papers.

Sort:

By area:

  1. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  2. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  3. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  4. 2025WildIFEval: Instruction Following in the Wildcited by 11
  5. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 8
  6. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 7
  7. 2026Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Whycited by 6
  8. 2025Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluationcited by 3
  9. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easycited by 3
  10. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judgescited by 3
  11. 2026RLPO: Residual Listwise Preference Optimization for Long-Context Review Rankingcited by 3
  12. 2025NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Taskscited by 2
17 more
  1. 2026When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judgescited by 2
  2. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agentscited by 2
  3. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challengescited by 1
  4. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  5. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 1
  6. 2026How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustnesscited by 1
  7. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 1
  8. 2026Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoningcited by 1
  9. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  10. 2026Aligning Language Model Benchmarks with Pairwise Preferencescited by 0
  11. 2026Suffix-Constrained Greedy Search Algorithms for Causal Language Modelscited by 0
  12. 2026Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domainscited by 0
  13. 2026PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalizationcited by 0
  14. 2026Assessing the Downstream Utility of Evidence-Aware Retrieval in RAGcited by 0
  15. 2026PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoningcited by 0
  16. 2026Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurementcited by 0
  17. 2026StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judgecited by 0