mlsanityData to 14 Oct 2026

MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting

arXiv 2601.09085 ↗First posted 14 Jan 2026cs.LG cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has become a standard approach for training mathematical reasoning models; however, its reliance on multiple completions per prompt makes training computationally expensive. Although recent work has reduced the number of training steps required to reach peak performance, the overall wall-clock training time often remains unchanged or even increases due to higher per-step cost. We propose MMR-GRPO, which integrates Maximal Marginal Relevance to reweigh rewards based on completion diversity. Our key insight is that semantically redundant completions contribute limited marginal learning signal; prioritizing diverse solutions yields more informative updates and accelerates convergence. Extensive evaluations across three model sizes (1.5B, 7B, 8B), three GRPO variants, and five mathematical reasoning benchmarks show that MMR-GRPO achieves comparable peak performance while requiring on average 47.9% fewer training steps and 70.2% less wall-clock time. These gains are consistent across models, methods, and benchmarks. Our code is released at: https://github.com/WeiKangda/MMR-GRPO.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 43 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

33 of 43 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement8 Dec
  2. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation8 Oct
  3. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning1 Oct
  4. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  5. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization23 Sep
  6. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models24 Jul
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models5 Jul
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  9. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization18 May
  10. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning14 May
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
4 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  9. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  10. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  11. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  12. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 72
5 more
  1. 2020The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Researchcited by 67
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  4. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
  5. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 7
  2. 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
  2. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  3. 2026GRPO Training Dynamics for Small Language Models
  4. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  5. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  6. 2026ReCo: Reweighting GRPO Against Distributional Concentration
  7. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning