mlsanityData to 14 Oct 2026

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

arXiv 2504.15253 ↗First posted 21 Apr 2025cs.CL cs.LGcited by 44

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Scaling test-time computation, or affording a generator large language model (LLM) extra compute during inference, typically employs the help of external non-generative evaluators (i.e., reward models). Concurrently, LLM-judges, models trained to generate evaluations and critiques (explanations) in natural language, are becoming increasingly popular in automatic evaluation. Despite judge empirical successes, their effectiveness as evaluators in test-time scaling settings is largely unknown. In this paper, we introduce the Judge Evaluation for Test-Time Scaling (JETTS) benchmark, which evaluates judge performance in three domains (math reasoning, code generation, and instruction following) under three task settings: response reranking, step-level beam search, and critique-based response refinement. We evaluate 10 different judge models (7B-70B parameters) for 8 different base generator models (6.7B-72B parameters). Our benchmark shows that while judges are competitive with outcome reward models in reranking, they are consistently worse than process reward models in beam search procedures. Furthermore, though unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 55 of its 74 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

55 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 54

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
38 more
  1. 2024OpenAI o1 System Cardcited by 1,244
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  13. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  14. 2022Self-critiquing models for assisting human evaluatorscited by 380
  15. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  16. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  17. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  18. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  19. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  20. 2023Generative Judge for Evaluating Alignmentcited by 169
  21. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  22. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  23. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  24. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  25. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  26. 2024How to Evaluate Reward Models for RLHFcited by 88
  27. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
  28. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  29. 2024Self-Taught Evaluatorscited by 72
  30. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  31. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  32. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilitiescited by 44
  33. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  34. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  35. 2024Direct Judgement Preference Optimizationcited by 33
  36. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  37. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 32
  38. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 26.

44 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  2. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 33
  3. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  4. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 28
  5. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 24
  6. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 22
  7. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  8. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
  9. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 17
  10. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  11. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  12. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
32 more
  1. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  2. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  3. 2025Can You Trick the Grader? Adversarial Persuasion of LLM Judgescited by 9
  4. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  5. 2025SSR: Socratic Self-Refine for Large Language Model Reasoningcited by 7
  6. 2026MAS-ProVe: Understanding the Process Verification of Multi-Agent Systemscited by 6
  7. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
  8. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  9. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  10. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 4
  11. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  12. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generationcited by 3
  13. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  14. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  15. 2025Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Modelscited by 2
  16. 2026Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoningcited by 2
  17. 2026Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agentscited by 2
  18. 2025Using Span Queries to Optimize for Cache and Attention Localitycited by 1
  19. 2026DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generationcited by 1
  20. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Modelscited by 1
  21. 2026Permutation-Consensus Listwise Judging for Robust Factuality Evaluationcited by 1
  22. 2026An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Modelscited by 1
  23. 2026Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judgescited by 1
  24. 2026BudgetVerify: Budget-Tiered Verification for Financial QAcited by 1
  25. 2026ReCCur: A Recursive Corner-Case Curation Framework for Robust Vision-Language Understanding in Open and Edge Scenarioscited by 0
  26. 2026Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitionscited by 0
  27. 2026Joint Consistency: A Unified Test-Time Aggregation Framework via Energy Minimizationcited by 0
  28. 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scalingcited by 0
  29. 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verificationcited by 0
  30. 2026Sharding Prevents LLM Oversight Failures and Adversarial Exploitationcited by 0
  31. 2026Symphony for Text Generation: Benchmarking Clinical Note Generationcited by 0
  32. 2026VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Verdict: A Library for Scaling Judge-Time Compute
  2. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
  3. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
  4. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluators
  5. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  6. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols
  7. 2025Evaluating Scoring Bias in LLM-as-a-Judge
  8. 2026Codifying the Judge: Scalable Evaluation via Program Distillation