mlsanityData to 14 Oct 2026

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

arXiv 2609.35750 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 36 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

28 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents6 Jul

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  11. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  12. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
15 more
  1. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  2. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  5. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
  6. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  7. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 32
  8. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24
  9. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  10. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 19
  11. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  12. 2026Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RLcited by 12
  13. 2026Self-Compacting Language Model Agentscited by 11
  14. 2026Parallel Context Compaction for Long-Horizon LLM Agent Servingcited by 5
  15. 2026Prefix Sliding for efficient test-time scalingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Visual Memory Attacks Can Persist Through The KV Cachecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
  2. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning
  3. 2026LOCAL: Enabling Learning On-device Contiguously for Agent LLMs
  4. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
  5. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
  6. 2026KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems
  7. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
  8. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse