mlsanityData to 14 Oct 2026

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

arXiv 2603.01571 ↗First posted 2 Mar 2026cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at hrefhttps://huggingface.co/collections/DonJoey/mix-grmHugging Face, and the code is released at hrefhttps://github.com/Don-Joey/Mix-GRMGithub.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 66 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

57 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025RM-R1: Reward Modeling as Reasoning5 May
  2. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  7. 2023Let's Verify Step by Step31 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
37 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  2. 2024Self-Rewarding Language Modelscited by 681
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  5. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  9. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  10. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  11. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  12. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  13. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  14. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  15. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  16. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  17. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  18. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  19. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  20. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  21. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  22. 2024How to Evaluate Reward Models for RLHFcited by 88
  23. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  24. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  25. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  26. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  27. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  28. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  29. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  30. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilitiescited by 44
  31. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  32. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  33. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  34. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 19
  35. 2025Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verificationcited by 17
  36. 2024SCAR: Data Selection via Style Consistency-Aware Response Ranking for Efficient Instruction-Tuning of Large Language Modelscited by 6
  37. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Self-Evolving Deep Research via Joint Generation and Evaluationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
  2. 2025When More is Less: Understanding Chain-of-Thought Length in LLMs
  3. 2025Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought
  4. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  5. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
  6. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
  7. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
  8. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning