# ZipRL: Adaptive Multi-Turn Context Compression with Hindsight Response Replay

arXiv: https://arxiv.org/abs/2605.28069 · HTML: https://mlsanity.com/paper/2605.28069/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 25 of 25)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [AgentFold: Long-Horizon Web Agents with Proactive Context Management](https://mlsanity.com/paper/2510.24699/) (2025)
- [Let's reward step by step: Step-Level reward model as the Navigators for Reasoning](https://mlsanity.com/paper/2310.10080/) (2023)
- [Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS](https://mlsanity.com/paper/2411.18478/) (2024)
- [SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents](https://mlsanity.com/paper/2509.06283/) (2025)
- [CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2509.25004/) (2025)
- [Atlas: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning](https://mlsanity.com/paper/2601.03872/) (2026)
- [SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression](https://mlsanity.com/paper/2507.05633/) (2025)
- [Recurrent Context Compression: Efficiently Expanding the Context Window of LLM](https://mlsanity.com/paper/2406.06110/) (2024)
- [Enhancing RAG Efficiency with Adaptive Context Compression](https://mlsanity.com/paper/2507.22931/) (2025)
- [WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection](https://mlsanity.com/paper/2510.18798/) (2025)
- [Nested Browser-Use Learning for Agentic Information Seeking](https://mlsanity.com/paper/2512.23647/) (2025)
- [AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.17486/) (2025)

## Built on by (most cited first, 3 of 3)

- [TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs](https://mlsanity.com/paper/2609.33589/) (2026)
- [Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning](https://mlsanity.com/paper/2609.34422/) (2026)
- [SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation](https://mlsanity.com/paper/2609.36601/) (2026)

## Abstract (arXiv)

Adaptive context compression is vital for scaling Large Language Models (LLMs) to complex, multi-turn agent tasks. However, rule-based compression methods may discard task-critical nuances, while Reinforcement Learning (RL) approaches usually struggle to balance information retention and token efficiency under the sparse rewards inherent to long-horizon workflows. To bridge this gap, we propose ZipRL, a novel adaptive compression framework tailored for Reinforcement Learning from Verifiable Rewards (RLVR). ZipRL features a multi-granularity compression mechanism for active, non-uniform information reduction, coupled with Hindsight Response Replay (HRR), a technique designed to densify training signals during RLVR optimization. Theoretically, we prove ZipRL's superior task-relevant utility over uniform methods. Concretely, ZipRL utilizes coarse-to-fine prompts for macro-compression and incorporates HRR into GRPO via generalized advantage reshaping. Multiple models of varying versions and parameter scales validate the effectiveness of our approach. Benchmarks on five agent tasks show ZipRL outperforms state-of-the-art approaches by 27.9% and 34.7% across Qwen3-4B and Qwen3-8B models, while maintaining exceptional token efficiency and robustness under extreme 256-turn extrapolation stress tests.