# HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

arXiv: https://arxiv.org/abs/2606.03768 · HTML: https://mlsanity.com/paper/2606.03768/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [LightThinker: Thinking Step-by-Step Compression](https://mlsanity.com/paper/2502.15589/) (2025)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator](https://mlsanity.com/paper/2412.12094/) (2024)
- [Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios](https://mlsanity.com/paper/2409.19272/) (2024)
- [Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought](https://mlsanity.com/paper/2512.21711/) (2025)
- [GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment](https://mlsanity.com/paper/2505.12215/) (2025)
- [MEMENTO: Teaching LLMs to Manage Their Own Context](https://mlsanity.com/paper/2604.09852/) (2026)
- [Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors](https://mlsanity.com/paper/2510.08907/) (2025)
- [COMI: Coarse-to-fine Context Compression via Marginal Information Gain](https://mlsanity.com/paper/2602.01719/) (2026)
- [In-Context Former: Lightning-fast Compressing Context for Large Language Model](https://mlsanity.com/paper/2406.13618/) (2024)
- [Read As Human: Compressing Context via Parallelizable Close Reading and Skimming](https://mlsanity.com/paper/2602.01840/) (2026)
- [Hold Onto That Thought: Assessing KV Cache Compression On Reasoning](https://mlsanity.com/paper/2512.12008/) (2025)
- [CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling](https://mlsanity.com/paper/2602.01766/) (2026)
- [LightThinker++: From Reasoning Compression to Memory Management](https://mlsanity.com/paper/2604.03679/) (2026)

## Abstract (arXiv)

Extended chain-of-thought (CoT) traces improve LLM reasoning but incur substantial computational and memory costs. While existing CoT compression methods mitigate this by condensing thought steps into compact representations via memory tokens and retaining only these representations at inference time, the loss of fine-grained information makes subsequent steps more error-prone. To alleviate this, we propose HybridThinker, where in addition to preserved these representations, thought steps are also temporarily retained to provide fine-grained details. However, we observe that naively keeping thought steps accessible to subsequent steps during training lets the model bypass memory tokens by retrieving information directly from these steps, leaving the model's ability to compress and retrieve information through memory tokens insufficiently trained. We therefore introduce a hybrid training scheme, in which only some thought steps are directly accessible through attention to subsequent steps, while the other thought steps are masked, forcing the model to use memory tokens for compression and retrieval. Across 4 reasoning benchmarks, HybridThinker matches the uncompressed baseline, advancing the state of the art in CoT compression by 5.8 points on average accuracy with similar inference time. Ablation studies confirm that both temporary thought-step retention and the hybrid training scheme contribute to these gains.