# Improve LLM-as-a-Judge Ability as a General Ability

arXiv: https://arxiv.org/abs/2502.11689 · HTML: https://mlsanity.com/paper/2502.11689/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [HelpSteer2: Open-source dataset for training top-performing reward models](https://mlsanity.com/paper/2406.08673/) (2024)
- [How to Evaluate Reward Models for RLHF](https://mlsanity.com/paper/2410.14872/) (2024)
- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution](https://mlsanity.com/paper/2410.16256/) (2024)
- [Beyond Scalar Reward Model: Learning Generative Judge from Preference Data](https://mlsanity.com/paper/2410.03742/) (2024)

## Built on by (most cited first, 30 of 37)

- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning](https://mlsanity.com/paper/2505.10320/) (2025)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [R3: Robust Rubric-Agnostic Reward Models](https://mlsanity.com/paper/2505.13388/) (2025)
- [Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards](https://mlsanity.com/paper/2505.02686/) (2025)
- [CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards](https://mlsanity.com/paper/2507.09104/) (2025)
- [WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality](https://mlsanity.com/paper/2510.18560/) (2025)
- [J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization](https://mlsanity.com/paper/2505.13346/) (2025)
- [Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty](https://mlsanity.com/paper/2604.10072/) (2026)
- [J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge](https://mlsanity.com/paper/2505.11875/) (2025)
- [S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models](https://mlsanity.com/paper/2509.22099/) (2025)
- [Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains](https://mlsanity.com/paper/2510.17793/) (2025)
- [A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models](https://mlsanity.com/paper/2508.12903/) (2025)
- [Reward Model Routing in Alignment](https://mlsanity.com/paper/2510.02850/) (2025)
- [IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation](https://mlsanity.com/paper/2511.01014/) (2025)
- [Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance](https://mlsanity.com/paper/2605.07461/) (2026)
- [CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria](https://mlsanity.com/paper/2601.20327/) (2026)
- [Long-form RewardBench: Evaluating Reward Models for Long-form Generation](https://mlsanity.com/paper/2603.12963/) (2026)
- [FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models](https://mlsanity.com/paper/2605.10141/) (2026)
- [Do Biased Models Have Biased Thoughts?](https://mlsanity.com/paper/2508.06671/) (2025)
- [DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seeking](https://mlsanity.com/paper/2602.00238/) (2026)
- [Learning Ordinal Probabilistic Reward from Preferences](https://mlsanity.com/paper/2602.12660/) (2026)
- [ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework](https://mlsanity.com/paper/2604.07506/) (2026)
- [BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detection](https://mlsanity.com/paper/2604.11121/) (2026)
- [Exploring the features used for summary evaluation by Human and GPT](https://mlsanity.com/paper/2512.19620/) (2025)
- [Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge](https://mlsanity.com/paper/2603.04417/) (2026)
- [LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspection](https://mlsanity.com/paper/2604.05371/) (2026)
- [REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak](https://mlsanity.com/paper/2605.20654/) (2026)
- [Towards Spec Learning: Inference-Time Alignment from Preference Pairs](https://mlsanity.com/paper/2606.24004/) (2026)
- [TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMs](https://mlsanity.com/paper/2607.11258/) (2026)

## Abstract (arXiv)

LLM-as-a-Judge leverages the generative and reasoning capabilities of large language models (LLMs) to evaluate LLM responses across diverse scenarios, providing accurate preference signals. This approach plays a vital role in aligning LLMs with human values, ensuring ethical and reliable AI outputs that align with societal norms. Recent studies have raised many methods to train LLM as generative judges, but most of them are data consuming or lack accuracy, and only focus on LLM's judge ability. In this work, we regard judge ability as a general ability of LLM and implement a two-stage training approach, comprising supervised fine-tuning (SFT) warm-up and direct preference optimization (DPO) enhancement, to achieve judge style adaptation and improve judgment accuracy. Additionally, we introduce an efficient data synthesis method to generate judgmental content. Experimental results demonstrate that our approach, utilizing only about 2% to 40% of the data required by other methods, achieves SOTA performance on RewardBench. Furthermore, our training method enhances the general capabilities of the model by constructing complicated judge task, and the judge signals provided by our model have significantly enhanced the downstream DPO training performance of our internal models in our test to optimize policy model with Judge Model. We also open-source our model weights and training data to facilitate further research.