TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 38 references that are in the map. First posted on arXiv 26 Aug 2026.
Builds on
36 of 38 references.
Closest ancestors · 9
Same direction, newest first
- 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective17 Mar
- 2025Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents16 Dec
- 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
- 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems14 Oct
- 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
- 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
- 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving5 Nov
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
- 2023ChatDev: Communicative Agents for Software Development16 Jul
Other references · 27
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
15 more
- 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
- 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
- 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
- 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
- 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
- 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
- 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
- 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 15
- 2025Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloudcited by 11
- 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
- 2025FlowMesh: A Service Fabric for Composable LLM Workflowscited by 6
- 2025AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Servingcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
- 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
- 2026HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
- 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
- 2025Accelerating Language Model Workflows with Prompt Choreography
- 2026[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows
- 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
- 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs