mlsanityData to 7 Oct 2026

LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning

arXiv 2510.01459 ↗First posted 1 Oct 2025cs.LG cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Since the release of Deepseek-R1, reinforcement learning with verifiable rewards (RLVR) has become a central approach for training large language models (LLMs) on reasoning tasks. Recent work has largely focused on modifying loss functions to make RLVR more efficient and effective. In this paper, motivated by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects training data at each step based on the average response length. We evaluate LSPO across multiple base models and datasets, demonstrating that it consistently improves learning effectiveness. In addition, we conduct a detailed ablation study to examine alternative ways of incorporating length signals into dynamic sampling, offering further insights and highlighting promising directions for future research.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 41 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

36 of 41 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025LIMR: Less is More for RL Scaling17 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  9. 2023Let's Verify Step by Step31 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2025Group Sequence Policy Optimizationcited by 788
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
14 more
  1. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  2. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  3. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  4. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  5. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  6. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 86
  7. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 79
  8. 2025Concise Reasoning via Reinforcement Learningcited by 78
  9. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RLcited by 25
  10. 2024Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yetcited by 22
  11. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signalscited by 17
  12. 2025LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimizationcited by 11
  13. 2025Iterative Deepening Sampling as Efficient Test-Time Scalingcited by 8
  14. 2024Flaming-hot Initiation with Regular Execution Sampling for Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Your Group-Relative Advantage Is Biasedcited by 32
  2. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 11
  3. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  4. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  5. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 2
  6. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  7. 2026Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Constructioncited by 0
  8. 2026V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimizationcited by 0
  9. 2026QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
  2. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  3. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  4. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  5. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning
  6. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce