mlsanityData to 7 Oct 2026

It Takes Two: Your GRPO Is Secretly DPO

arXiv 2510.00977 ↗First posted 1 Oct 2025cs.LG cs.CLcited by 25

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the value baselines from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains 97.6% of the performance of 16-GRPO, while requiring only 12.5% of the rollouts and 21% of the training time.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 54 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

40 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Geometric-Mean Policy Optimization28 Jul
  2. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning10 Jun
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  4. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization18 May
  5. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
19 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2025Group Sequence Policy Optimizationcited by 788
  6. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  8. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  9. 2019Analyzing and Improving Representations with the Soft Nearest Neighbor Losscited by 95
  10. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  11. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  12. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  13. 2025REARANK: Reasoning Re-ranking Agent via Reinforcement Learningcited by 24
  14. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  15. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  16. 2025The Hidden Link Between RLHF and Contrastive Learningcited by 5
  17. 2025Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learningcited by 3
  18. 2022Momentum Contrastive Pre-training for Question Answeringcited by 2
  19. 2024Unifying Graph Convolution and Contrastive Learning in Collaborative Filteringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 23.

25 papers.

Sort:

By area:

  1. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 10
  2. 2025DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimizationcited by 8
  3. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learningcited by 6
  4. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  5. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 5
  6. 2026On the Plasticity and Stability for Post-Training Large Language Modelscited by 3
  7. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancementcited by 2
  8. 2026Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversionscited by 2
  9. 2026Interactive Critique-Revision Training for Reliable Structured LLM Generationcited by 2
  10. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 2
  11. 2026Rethinking Groups in Critic-Free RLVRcited by 2
  12. 2026GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identitycited by 2
13 more
  1. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language modelscited by 2
  2. 2026ReNCE: Learning to Reason by Noise Contrastive Estimationcited by 1
  3. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Modelscited by 1
  4. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  5. 2026BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimizationcited by 1
  6. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  7. 2026Constraint-Rectified Training for Efficient Chain-of-Thoughtcited by 0
  8. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  9. 2026How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVRcited by 0
  10. 2026Spurious Advantage Hidden in GRPOcited by 0
  11. 2026Inducing Emergent Misalignment from Reward Hacks with Iterative DPOcited by 0
  12. 2026Direct Preference Density Alignment for Conversational Audio Equalizationcited by 0
  13. 2026Agentic Multi-Turn Reasoning: A Fairness Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  2. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  3. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  4. 2026ReCo: Reweighting GRPO Against Distributional Concentration
  5. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  6. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  7. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  8. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO