mlsanityData to 7 Oct 2026

Improve LLM-as-a-Judge Ability as a General Ability

arXiv 2502.11689 ↗First posted 17 Feb 2025cs.CLcited by 35

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

LLM-as-a-Judge leverages the generative and reasoning capabilities of large language models (LLMs) to evaluate LLM responses across diverse scenarios, providing accurate preference signals. This approach plays a vital role in aligning LLMs with human values, ensuring ethical and reliable AI outputs that align with societal norms. Recent studies have raised many methods to train LLM as generative judges, but most of them are data consuming or lack accuracy, and only focus on LLM's judge ability. In this work, we regard judge ability as a general ability of LLM and implement a two-stage training approach, comprising supervised fine-tuning (SFT) warm-up and direct preference optimization (DPO) enhancement, to achieve judge style adaptation and improve judgment accuracy. Additionally, we introduce an efficient data synthesis method to generate judgmental content. Experimental results demonstrate that our approach, utilizing only about 2% to 40% of the data required by other methods, achieves SOTA performance on RewardBench. Furthermore, our training method enhances the general capabilities of the model by constructing complicated judge task, and the judge signals provided by our model have significantly enhanced the downstream DPO training performance of our internal models in our test to optimize policy model with Judge Model. We also open-source our model weights and training data to facilitate further research.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 32 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

19 of 32 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  12. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
7 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  4. 2024How to Evaluate Reward Models for RLHFcited by 84
  5. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  6. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  7. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 19.

35 papers.

Sort:

By area:

  1. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  2. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  3. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  4. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  6. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11
  7. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 10
  8. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  9. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  10. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  11. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  12. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
23 more
  1. 2025A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Modelscited by 4
  2. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  3. 2025Reward Model Routing in Alignmentcited by 3
  4. 2026Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidancecited by 3
  5. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  6. 2026FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Modelscited by 2
  7. 2025Do Biased Models Have Biased Thoughts?cited by 1
  8. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 1
  9. 2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
  10. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Frameworkcited by 1
  11. 2026BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detectioncited by 1
  12. 2025Exploring the features used for summary evaluation by Human and GPTcited by 0
  13. 2026DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seekingcited by 0
  14. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judgecited by 0
  15. 2026LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspectioncited by 0
  16. 2026REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreakcited by 0
  17. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  18. 2026TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMscited by 0
  19. 2026Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utilitycited by 0
  20. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0
  21. 2026SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decompositioncited by 0
  22. 2026HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applicationscited by 0
  23. 2026BizSage: A Self-Evolving Multi-Agent Framework for Business Research with Efficient Knowledge Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge
  2. 2024Direct Judgement Preference Optimization
  3. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  4. 2025Reverse Engineering Human Preferences with Reinforcement Learning
  5. 2025JudgeLRM: Large Reasoning Models as a Judge
  6. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniques
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  8. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments