mlsanityData to 7 Oct 2026

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

arXiv 2505.12366 ↗First posted 18 May 2025cs.LG cs.AIcited by 16

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7% over GRPO and 6% over DAPO across six benchmark tasks for a 1.5B model.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 85 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

48 of 85 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  3. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  4. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning6 Apr
  5. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  7. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
4 more
  1. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  3. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  12. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
20 more
  1. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  2. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  4. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  5. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  6. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  7. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  8. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  9. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  10. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  11. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  12. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  13. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  14. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  15. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  16. 2016Minimizing the Maximal Loss: How and Why?cited by 30
  17. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  18. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  19. 2024A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safetycited by 3
  20. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 25
  2. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  3. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  4. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 13
  5. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12
  6. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  7. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  8. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspectivecited by 4
  9. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  10. 2025Accelerating RLHF Training with Reward Variance Increasecited by 2
  11. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  12. 2026ReSS: Learning Reasoning Models for Tabular Data Prediction via Symbolic Scaffoldcited by 1
4 more
  1. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Modelscited by 1
  2. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspectivecited by 1
  3. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
  4. 2026Retrospective Distillation Attribution via Normalized Response Similaritycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  2. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  3. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
  4. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  5. 2026Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
  6. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
  7. 2025DARO: Difficulty-Aware Reweighting Policy Optimization
  8. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO