mlsanityData to 14 Oct 2026

Mitigating the Length-Scaling Tax with Online Distillation

arXiv 2609.38854 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning also fits: Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Length scaling during reinforcement-learning (RL) post-training is often viewed as a sign of improved reasoning ability, especially on difficult problems, but may also make responses to already-solved problems unnecessarily verbose. We quantify this side effect as the length-scaling tax (LST): excess response length on already-solved queries without a commensurate accuracy gain. To mitigate LST, we propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation and retains the original RL objective for unsolved prompts. LSD uses an exponential moving average of the online policy as its teacher, requiring no external model. We find that LSD achieves comparable or better performance than RL across multiple variants, while substantially curbing response-length growth on easy queries. LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, demonstrating that LSD effectively preserves concise response patterns on easy queries while supporting efficient exploration on difficult queries during RL post-training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

25 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning24 Feb
  2. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models2 Feb
  3. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2023Let's Verify Step by Step31 May

Other references · 20

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  12. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
8 more
  1. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  2. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 158
  3. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 72
  4. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  5. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learningcited by 52
  6. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RLcited by 25
  7. 2026Contrastive On-Policy Distillationcited by 3
  8. 2026The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Optimal Reasoning Length for RL-Trained Language Models
  2. 2026To Think or Not to Think: Allocating Reasoning Where It Helps
  3. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
  4. 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model
  5. 2025Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
  6. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
  7. 2026LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models
  8. 2026Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision