Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Chain-of-thought reasoning in large language models can trigger an "overthinking trap": longer rollouts raise cost and latency yet often yield unreliable accuracy gains. Existing methods use global, static controls that may suppress needed reasoning. We propose mastery-gated, sample-level, soft reinforcement learning compression that penalizes long rollouts only when the model already solves the problem and has produced a shorter rollout. Across benchmarks, it cuts response length by 20-40% with comparable or higher accuracy and generalizes across domains: a model trained on math spontaneously shortens unseen tasks (code, instruction following, general-knowledge QA) without hurting accuracy. We further show two-way transfer between non-agent CoT and tool-use agents: non-agent training reduces SWE-Bench Verified rounds by 13%, while compressing a thinking agent cuts SWE trajectories by 67% tokens and 52% rounds and shortens non-agent outputs by up to 44%. Compression is thus not cosmetic brevity, but an inherent computation policy -- what to keep, and what to forget.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 35 references that are in the map. First posted on arXiv 19 Dec 2025.
Builds on
17 of 35 references.
Closest ancestors · 7
Same direction, newest first
- 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning16 Oct
- 2025The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging26 Sep
- 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning20 May
- 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning2 Apr
- 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging26 Mar
- 2025LightThinker: Thinking Step-by-Step Compression21 Feb
- 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning22 Jan
Other references · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2025Group Sequence Policy Optimizationcited by 788
- 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
- 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillationcited by 39
- 2025Efficient Long CoT Reasoning in Small Language Modelscited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
- 2025Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
- 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
- 2026ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning
- 2026Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
- 2026Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization
- 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
- 2026Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains