# ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression

arXiv: https://arxiv.org/abs/2605.07501 · HTML: https://mlsanity.com/paper/2605.07501/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 32)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Learn to Reason Efficiently with Adaptive Length-based Reward Shaping](https://mlsanity.com/paper/2505.15612/) (2025)
- [Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL](https://mlsanity.com/paper/2505.10832/) (2025)
- [Efficiently Scaling LLM Reasoning with Certaindex](https://mlsanity.com/paper/2412.20993/) (2024)
- [Optimizing Length Compression in Large Reasoning Models](https://mlsanity.com/paper/2506.14755/) (2025)
- [DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning](https://mlsanity.com/paper/2510.15110/) (2025)
- [Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning](https://mlsanity.com/paper/2504.04383/) (2025)
- [XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation](https://mlsanity.com/paper/2510.06672/) (2025)
- [Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model](https://mlsanity.com/paper/2512.21540/) (2025)
- [Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.21919/) (2026)
- [STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization](https://mlsanity.com/paper/2511.13091/) (2025)
- [WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning](https://mlsanity.com/paper/2602.17025/) (2026)
- [Learning to Hint for Reinforcement Learning](https://mlsanity.com/paper/2604.00698/) (2026)
- [F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare](https://mlsanity.com/paper/2602.06717/) (2026)
- [Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking](https://mlsanity.com/paper/2509.23392/) (2025)
- [Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning](https://mlsanity.com/paper/2602.20722/) (2026)
- [Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning](https://mlsanity.com/paper/2603.10535/) (2026)
- [LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2510.16552/) (2025)
- [DART: Difficulty-Adaptive Reasoning Truncation for Efficient Large Language Models](https://mlsanity.com/paper/2511.01170/) (2025)
- [Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning](https://mlsanity.com/paper/2602.22642/) (2026)
- [Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization](https://mlsanity.com/paper/2601.06052/) (2025)
- [Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2603.00296/) (2026)
- [Efficient Reasoning via Thought-Training and Thought-Free Inference](https://mlsanity.com/paper/2511.03408/) (2025)
- [RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization](https://mlsanity.com/paper/2601.19404/) (2026)

## Built on by (most cited first, 2 of 2)

- [Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data](https://mlsanity.com/paper/2606.05122/) (2026)
- [Learning Multimodal Embeddings with Evidence-Aligned Readout](https://mlsanity.com/paper/2609.33659/) (2026)

## Abstract (arXiv)

Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency. Existing reinforcement learning (RL) approaches for CoT compression rely on uniform, static length penalties that neglect model capability dynamics and problem-level difficulty variation. We propose ExpThinkxspace, an RL framework that addresses both dimensions through two complementary mechanisms. First, experience-guided reward shaping tracks the shortest correct solution found so far for each problem and applies a three-tier reward: full credit for concise correct responses, discounted credit for verbose correct ones, and zero for incorrect ones. The threshold tightens automatically with model improvement, forming a self-evolving curriculum that requires no manual scheduling. Second, difficulty-adaptive advantage replaces standard deviation normalization with correct-count normalization, yielding monotonically difficulty-scaled gradients that amplify learning on hard problems to preserve accuracy while suppressing gradients on easy ones to encourage brevity. Together, these mechanisms enforce an accuracy-first, compression-second training objective. Experiments on multiple mathematical reasoning benchmarks demonstrate that ExpThinkxspace reduces average response length by up to 77% while simultaneously improving accuracy, achieving up to 3× higher accuracy-efficiency ratio (accuracy divided by average token count) than the vanilla baseline and outperforming existing RL-based compression methods on both metrics.