mlsanityData to 7 Oct 2026

From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks

arXiv 2609.37145 ↗First posted 29 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code also fits: Reinforcement learning for reasoning

Abstract (arXiv)

LLM-as-a-Judge is increasingly used to evaluate policy responses on open-ended tasks that lack ground-truth answers. Existing work often directly converts the resulting judgments into reward signals for policy training, paying limited attention to intrinsic judgment quality and largely restricting the use of Judges to training-time supervision. We systematically investigate judgment quality and downstream utility by examining both how judgments are elicited and how they are used. For judgment elicitation, we vary the Judge protocol along three dimensions: verdict granularity, critique usage, and evaluation batching. For judgment usage, beyond policy training, we extend Judge to test-time inference through Best-of-N selection, Judge-guided revision, and beam search. We find that, (i) Surprisingly, judgment quality and downstream utility do not always align. (ii) Judge protocol design substantially affects both intrinsic judgment quality and downstream utility. (iii) Judge guidance effectively converts test-time compute into performance gains, with benefits varying across inference strategies. Our results call for a multifaceted evaluation of LLM Judges on open-ended tasks, encompassing intrinsic judgment quality, and downstream utility.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 17 of its 31 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

17 of 31 references.

References · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  3. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  4. 2025Reinforcement Learning with Rubric Anchorscited by 108
  5. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  6. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  7. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  8. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  9. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  10. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?cited by 23
  11. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 23
  12. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
5 more
  1. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 11
  2. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubriccited by 8
  3. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 5
  4. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  5. 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judgecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
  2. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
  3. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
  4. 2026Codifying the Judge: Scalable Evaluation via Program Distillation
  5. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
  6. 2026RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
  7. 2025Improving LLM-as-a-Judge Inference with the Judgment Distribution
  8. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization