# Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management

arXiv: https://arxiv.org/abs/2510.06727 · HTML: https://mlsanity.com/paper/2510.06727/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents](https://mlsanity.com/paper/2509.13309/) (2025)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion](https://mlsanity.com/paper/2501.15089/) (2025)
- [PENCIL: Long Thoughts with Short Memory](https://mlsanity.com/paper/2503.14337/) (2025)
- [MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use](https://mlsanity.com/paper/2508.18669/) (2025)
- [Cognitive Memory in Large Language Models](https://mlsanity.com/paper/2504.02441/) (2025)
- [Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://mlsanity.com/paper/2408.01866/) (2024)

## Built on by (most cited first, 30 of 51)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Scaling Long-Horizon LLM Agent via Context-Folding](https://mlsanity.com/paper/2510.11967/) (2025)
- [Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2510.12635/) (2025)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [Benchmark Test-Time Scaling of General LLM Agents](https://mlsanity.com/paper/2602.18998/) (2026)
- [AgentOCR: Reimagining Agent History via Optical Self-Compression](https://mlsanity.com/paper/2601.04786/) (2026)
- [Scaling Test-Time Compute for Agentic Coding](https://mlsanity.com/paper/2604.16529/) (2026)
- [SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents](https://mlsanity.com/paper/2606.28434/) (2026)
- [CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents](https://mlsanity.com/paper/2607.05378/) (2026)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory](https://mlsanity.com/paper/2603.04257/) (2026)
- [LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents](https://mlsanity.com/paper/2605.05191/) (2026)
- [SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent](https://mlsanity.com/paper/2605.24468/) (2026)
- [ACM: Agentic Context Management for Long Horizon Tasks](https://mlsanity.com/paper/2607.23809/) (2026)
- [Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability](https://mlsanity.com/paper/2608.06503/) (2026)
- [PABU: Progress-Aware Belief Update for Efficient LLM Agents](https://mlsanity.com/paper/2602.09138/) (2026)
- [FoldAct: Efficient and Stable Context Folding for Long-Horizon Search Agents](https://mlsanity.com/paper/2512.22733/) (2025)
- [Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents](https://mlsanity.com/paper/2605.08580/) (2026)
- [CoMem: Context Management with A Decoupled Long-Context Model](https://mlsanity.com/paper/2605.30842/) (2026)
- [Scaling Self-Evolving Agents via Parametric Memory](https://mlsanity.com/paper/2606.04536/) (2026)
- [CoACT: Action-Preserving Observation Compression for Coding Agents](https://mlsanity.com/paper/2607.02911/) (2026)
- [ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgetting](https://mlsanity.com/paper/2605.03804/) (2026)
- [Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents](https://mlsanity.com/paper/2606.11680/) (2026)
- [ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL](https://mlsanity.com/paper/2606.31650/) (2026)
- [Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning](https://mlsanity.com/paper/2605.15315/) (2026)
- [Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap](https://mlsanity.com/paper/2605.24432/) (2026)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [A Task-State Representation for Long-Horizon Mobile GUI Agents](https://mlsanity.com/paper/2607.00502/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)

## Abstract (arXiv)

We study reinforcement learning (RL) fine-tuning of large language model (LLM) agents for long-horizon multi-turn tool use, where context length quickly becomes a fundamental bottleneck. Existing RL pipelines can suffer from degraded instruction following, excessive rollout costs, and most importantly, strict context limits. To address these challenges, we introduce summarization-based context management to training. In specific, it periodically compresses the tool using history by LLM-generated summaries that retain task-relevant information to keep a compact context while enabling the agent to scale beyond the fixed context window. Building on this formulation, we derive a policy gradient representation that seamlessly enables standard LLM RL infrastructures to optimize both tool-use behaviors as well as summarization strategies in an end-to-end fashion. We instantiate this framework with underlineSUmmarization augmented underlinePolicy underlineOptimization (textttSUPO), an LLM RL algorithm that enables long-horizon training beyond a fixed context limit. Experiments on interactive function calling and searching tasks demonstrate that textttSUPO significantly improves the success rate while maintaining the same or even lower working context length compared to baselines. We also demonstrate that for complex searching tasks, textttSUPO can further improve the evaluation performance when scaling test-time maximum round of summarization beyond that of training time. Our results establish summarization-based context management as a principled and scalable approach for training RL agents beyond a fixed context length limit.