Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives. Existing multi-turn systems keep the turn as the scheduling unit and decide, turn by turn, whether to disaggregate prefill from decode. That decision rests on the turn's decode length, tool behavior, and KV growth, quantities that are not observable when the scheduler must act, forcing the system to predict them. We show this dependence on prediction is imposed by the scheduling unit, not the workload. Raising the scheduling unit from the turn to the conversation converts turn-level irregularity into a stable, two-phase structure: 1) a compute-bound turn-1 prefill followed by 2) a long, memory-bound tail. Thus, with the conversation as the scheduling unit, placement reduces to reading the first-turn input length and per-decoder KV occupancy, both directly observable. We instantiate this principle in ConServe, which routes the first-turn prefill to a high-throughput prefiller, transfers the KV cache exactly once, and pins the conversation to a single decoder for its entire tail, with no learned model of decode-side cost. Against a per-turn prediction baseline, ConServe reduces p95 time-to-first-effective-token (the latency of a conversation's first user-visible output) by 51.08% and improves energy efficiency by 7.51% while preserving last-turn TBT and SLOs; mapping the two phases onto heterogeneous GPU tiers adds a further 22.75% in energy efficiency.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 49 references that are in the map. First posted on arXiv 1 Jun 2026.
Builds on
33 of 49 references.
Closest ancestors · 3
Same direction, newest first
- 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
- 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
- 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
Other references · 30
- 2017Attention Is All You Needcited by 59,932
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
- 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
- 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
- 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
18 more
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
- 2024Efficient LLM Scheduling by Learning to Rankcited by 83
- 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
- 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
- 2024BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Cachingcited by 23
- 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
- 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
- 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
- 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
- 2024GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissionscited by 10
- 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
- 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancingcited by 7
- 2026Efficient Multi-round LLM Inference over Disaggregated Servingcited by 6
- 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
- 2026Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Servingcited by 5
- 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Stateful Inference for Low-Latency Multi-Agent Tool Calling
- 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
- 2025MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
- 2026Harnessing LLMs as Agents: What Does It Cost?
- 2026From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution
- 2025Towards Efficient Agents: A Co-Design of Inference Architecture and System
- 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
- 2026Second Thought: Reasoning in Parallel as LLM Agents Act and Observe