mlsanityData to 7 Oct 2026

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization

arXiv 2505.13346 ↗First posted 19 May 2025cs.CL cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

To keep pace with the increasing pace of large language models (LLM) development, model output evaluation has transitioned away from time-consuming human evaluation to automatic evaluation, where LLMs themselves are tasked with assessing and critiquing other model outputs. LLM-as-judge models are a class of generative evaluators that excel in evaluating relatively simple domains, like chat quality, but struggle in reasoning intensive domains where model responses contain more substantive and challenging content. To remedy existing judge shortcomings, we explore training judges with reinforcement learning (RL). We make three key contributions: (1) We propose the Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO) algorithm, which allows us to train our judge to be robust to positional biases that arise in more complex evaluation settings. (2) We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work. (3) We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%, matching or exceeding the performance of larger GRPO-trained judges on both JudgeBench and ReasoningJudgeBench.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 83 of its 96 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

83 of 96 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RM-R1: Reward Modeling as Reasoning5 May
  2. 2025Process Reward Models That Think23 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 74

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  14. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  15. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  16. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  17. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  18. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  19. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  20. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  21. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  22. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  23. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  24. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  25. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  26. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  27. 2023Generative Judge for Evaluating Alignmentcited by 163
  28. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  29. 2024Generative Reward Modelscited by 148
  30. 2024LLM Critics Help Catch LLM Bugscited by 138
  31. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  32. 2025BIG-Bench Extra Hardcited by 123
  33. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  34. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  35. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  36. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  37. 2023Shepherd: A Critic for Language Model Generationcited by 96
  38. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  2. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  3. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  4. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  5. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  6. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  7. 2025Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarizationcited by 0
  8. 2026Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judgescited by 0
  9. 2026Certified Selective Automation of LLM Agent Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
  2. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  3. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge
  4. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
  5. 2025Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
  6. 2024Aligning Large Language Models by On-Policy Self-Judgment
  7. 2026REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
  8. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation