# JudgeLRM: Large Reasoning Models as a Judge

arXiv: https://arxiv.org/abs/2504.00050 · HTML: https://mlsanity.com/paper/2504.00050/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2503.13939/) (2025)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)

## Built on by (most cited first, 30 of 80)

- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning](https://mlsanity.com/paper/2505.10320/) (2025)
- [Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training](https://mlsanity.com/paper/2602.01511/) (2026)
- [Reward Reasoning Model](https://mlsanity.com/paper/2505.14674/) (2025)
- [Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards](https://mlsanity.com/paper/2506.00103/) (2025)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [R3: Robust Rubric-Agnostic Reward Models](https://mlsanity.com/paper/2505.13388/) (2025)
- [Are We on the Right Way to Assessing LLM-as-a-Judge?](https://mlsanity.com/paper/2512.16041/) (2025)
- [Think-J: Learning to Think for Generative LLM-as-a-Judge](https://mlsanity.com/paper/2505.14268/) (2025)
- [ReviewRL: Towards Automated Scientific Review with RL](https://mlsanity.com/paper/2508.10308/) (2025)
- [Variation in Verification: Understanding Verification Dynamics in Large Language Models](https://mlsanity.com/paper/2509.17995/) (2025)
- [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025)
- [Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict](https://mlsanity.com/paper/2506.06485/) (2025)
- [AgentV-RL: Scaling Reward Modeling with Agentic Verifier](https://mlsanity.com/paper/2604.16004/) (2026)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training](https://mlsanity.com/paper/2603.12246/) (2026)
- [Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling](https://mlsanity.com/paper/2507.14783/) (2025)
- [Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation](https://mlsanity.com/paper/2512.22245/) (2025)
- [J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization](https://mlsanity.com/paper/2505.13346/) (2025)
- [ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models](https://mlsanity.com/paper/2506.16712/) (2025)
- [mR3: Multilingual Rubric-Agnostic Reward Reasoning Models](https://mlsanity.com/paper/2510.01146/) (2025)
- [Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization](https://mlsanity.com/paper/2603.08091/) (2026)
- [CHARM: Calibrating Reward Models With Chatbot Arena Scores](https://mlsanity.com/paper/2504.10045/) (2025)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Qworld: Question-Specific Evaluation Criteria for LLMs](https://mlsanity.com/paper/2603.23522/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.