mlsanityData to 14 Oct 2026

Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning

arXiv 2508.09726 ↗First posted 13 Aug 2025cs.CL cs.LGcited by 75

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models trained with reinforcement learning with verifiable rewards tend to trade accuracy for length--inflating response lengths to achieve gains in accuracy. While longer answers may be warranted for harder problems, many tokens are merely "filler": repetitive, verbose text that makes no real progress. We introduce GFPO (Group Filtered Policy Optimization), which curbs this length explosion by sampling larger groups per problem during training and filtering responses to train on based on two key metrics: (1) response length and (2) token efficiency: reward per token ratio. By sampling more at training time, we teach models to think less at inference time. On the Phi-4-reasoning model, GFPO cuts GRPO's length inflation by 46-71% across challenging STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH, LiveCodeBench) while maintaining accuracy. Optimizing for reward per token further increases reductions in length inflation to 71-85%. We also propose Adaptive Difficulty GFPO, which dynamically allocates more training resources to harder problems based on real-time difficulty estimates, improving the balance between computational efficiency and accuracy especially on difficult questions. GFPO demonstrates that increased training-time compute directly translates to reduced test-time compute--a simple yet effective trade-off for efficient reasoning.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 30 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

28 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  8. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  9. 2025Dynamic Early Exit in Reasoning Modelscited by 219
  10. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  11. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  12. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
10 more
  1. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  2. 2025Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoningcited by 85
  3. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  4. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 52
  5. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  6. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 44
  7. 2025Answer Convergence as a Signal for Early Stopping in Reasoningcited by 40
  8. 2025Optimizing Length Compression in Large Reasoning Modelscited by 38
  9. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  10. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewardscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 49.

75 papers.

Sort:

By area:

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  4. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  5. 2025Entropy After </Think> for reasoning model early exitingcited by 24
  6. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokenscited by 22
  7. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 19
  8. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  9. 2026MEMENTO: Teaching LLMs to Manage Their Own Contextcited by 18
  10. 2025Group Critical-token Policy Optimization for Autoregressive Image Generationcited by 17
  11. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  12. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learningcited by 14
38 more
  1. 2025Reinforcement Learning with Inverse Rewards for World Model Post-trainingcited by 13
  2. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoningcited by 13
  3. 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Modelcited by 13
  4. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  5. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  6. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  7. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  8. 2025Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexitycited by 9
  9. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  10. 2025Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoningcited by 9
  11. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  12. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  13. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  14. 2026Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoningcited by 7
  15. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 7
  16. 2025Reinforcement Mid-Trainingcited by 6
  17. 2025You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMscited by 6
  18. 2025How Does RL Post-training Induce Skill Composition? A Case Study on Countdowncited by 6
  19. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  20. 2026TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoningcited by 6
  21. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  22. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  23. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4
  24. 2026On the Optimal Reasoning Length for RL-Trained Language Modelscited by 4
  25. 2026What does RL improve for Visual Reasoning? A Frankenstein-Style Analysiscited by 4
  26. 2026LACONIC: Length-Aware Constrained Reinforcement Learning for LLMcited by 4
  27. 2026Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agentscited by 4
  28. 2025Concise Reasoning in the Lens of Lagrangian Optimizationcited by 3
  29. 2025Learning to Reason Efficiently with Discounted Reinforcement Learningcited by 3
  30. 2026ESTAR: Early-Stopping Token-Aware Reasoning For Efficient Inferencecited by 3
  31. 2026IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoningcited by 3
  32. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  33. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimizationcited by 3
  34. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  35. 2026SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Modelscited by 3
  36. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  37. 2026Masked Distillation: Internalizing the Chain-of-Thought in Language Modelscited by 3
  38. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVRcited by 2

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation
  2. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  3. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
  4. 2026Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization
  5. 2025Train Long, Think Short: Curriculum Learning for Efficient Reasoning
  6. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  8. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning