# Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

arXiv: https://arxiv.org/abs/2511.02230 · HTML: https://mlsanity.com/paper/2511.02230/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 60 of its 115 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 30 of 60)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)

## Built on by (most cited first, 30 of 44)

- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System](https://mlsanity.com/paper/2602.13692/) (2026)
- [TraceLab: Characterizing Coding Agent Workloads for LLM Serving](https://mlsanity.com/paper/2606.30560/) (2026)
- [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](https://mlsanity.com/paper/2601.22705/) (2026)
- [AgentCgroup: Understanding and Controlling OS Resources of AI Agents](https://mlsanity.com/paper/2602.09345/) (2026)
- [MEPIC: Memory Efficient Position Independent Caching for LLM Serving](https://mlsanity.com/paper/2512.16822/) (2025)
- [Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices](https://mlsanity.com/paper/2603.04428/) (2026)
- [Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving](https://mlsanity.com/paper/2604.25899/) (2026)
- [IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference](https://mlsanity.com/paper/2606.09916/) (2026)
- [Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference](https://mlsanity.com/paper/2601.12967/) (2026)
- [Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management](https://mlsanity.com/paper/2605.06472/) (2026)
- [IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents](https://mlsanity.com/paper/2605.22154/) (2026)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling](https://mlsanity.com/paper/2608.10402/) (2026)
- [Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning](https://mlsanity.com/paper/2604.05404/) (2026)
- [MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems](https://mlsanity.com/paper/2604.26963/) (2026)
- [Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches](https://mlsanity.com/paper/2605.18825/) (2026)
- [Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation](https://mlsanity.com/paper/2606.01725/) (2026)
- [Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving](https://mlsanity.com/paper/2606.02964/) (2026)
- [MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference](https://mlsanity.com/paper/2607.10582/) (2026)
- [A Policy-Driven Runtime Layer for Agentic LLM Serving](https://mlsanity.com/paper/2605.27744/) (2026)
- [Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving](https://mlsanity.com/paper/2606.01839/) (2026)
- [Practical Online KV Cache Compaction for LLM Agents: An Empirical Study](https://mlsanity.com/paper/2608.00902/) (2026)
- [Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation](https://mlsanity.com/paper/2608.26195/) (2026)
- [Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service](https://mlsanity.com/paper/2603.08739/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](https://mlsanity.com/paper/2604.16682/) (2026)
- [CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration](https://mlsanity.com/paper/2604.25080/) (2026)
- [QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs](https://mlsanity.com/paper/2605.03884/) (2026)
- [AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programs](https://mlsanity.com/paper/2606.09613/) (2026)

## Abstract (arXiv)

KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since many tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, many challenges remain. First, we need to consider both the potential cost of recomputation or reloading (if offloading enabled) as well as the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, the method needs to remain robust under limited predictability of tool call durations. We present Continnum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retention. For requests that generate tool calls, Continnum selectively pins the KV cache in GPU memory with a time-to-live value determined by the reload cost and potential queueing delay induced by eviction. When the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continnum preserves multi-turn continuity, and reduces delay for agentic workflows. Evaluations on real-world agents (SWE-Bench, BFCL, OpenHand) with Llama-3.1 8B/70B, Gemma-3 12B, and GLM-4.5 355B shows that Continnum improves the average job completion times by over 8x while improving throughput.