Data to 14 Oct 2026

J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

arXiv 2505.10320 ↗First posted 15 May 2025cs.CL cs.AI cs.LGcited by 75

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 45 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

39 of 45 references.

Closest ancestors · 4

Same direction, newest first

2025Qwen3 Technical Report14 May
2025JudgeLRM: Large Reasoning Models as a Judge31 Mar
2025Improve LLM-as-a-Judge Ability as a General Ability17 Feb
2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge30 Jan

Other references · 35

2022Training language models to follow instructions with human feedbackcited by 15,971
2024The Llama 3 Herd of Modelscited by 14,715
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2024GPT-4o System Cardcited by 4,588
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
2024A Survey on LLM-as-a-Judgecited by 1,423
2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210

23 more

2023Large Language Models are not Fair Evaluatorscited by 1,085
2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
2025RM-R1: Reward Modeling as Reasoningcited by 154
2024Generative Reward Modelscited by 148
2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
2024Critique-out-Loud Reward Modelscited by 106
2024How to Evaluate Reward Models for RLHFcited by 88
2024Self-Taught Evaluatorscited by 72
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
2024Improving Reward Models with Synthetic Critiquescited by 44
2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
2024Direct Judgement Preference Optimizationcited by 33
2024Self-rationalization improves LLM as a fine-grained judgecited by 27
2025Atla Selene Mini: A General Purpose Evaluation Modelcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 40.

75 papers.

Sort:

By area:

2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
2025Reward Reasoning Modelcited by 47
2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28

38 more

2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 20
2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 13
2025Learning to Refine: Self-Refinement of Parallel Reasoning in LLMscited by 12
2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
2025VRPRM: Process Reward Modeling via Visual Reasoningcited by 11
2025Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selectioncited by 9
2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8
2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 8
2025Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimizationcited by 7
2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
2025Self-Improving VLM Judges Without Human Annotationscited by 7
2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 7
2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
2025Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoningcited by 6
2025SCI-Verifier: Scientific Verifier with Thinkingcited by 6
2026Self-Improving Pretraining: using post-trained models to pretrain better modelscited by 6
2026Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluationcited by 6
2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
2026Debate Training Reduces Reward Hacking in RLAIFcited by 4
2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
2026MJ1: Multimodal Judgment via Grounded Verificationcited by 3
2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modelingcited by 3

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way