mlsanityData to 7 Oct 2026

SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters

arXiv 2605.00528 ↗First posted 1 May 2026cs.DC cs.AI cs.LGcited by 5

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

AI agents execute tens to hundreds of chained LLM calls per task, yet GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x. We argue that this request-level abstraction is fundamentally mismatched to compound AI workloads, and propose a shift to program-level scheduling: treating the entire agent workflow (not individual inference calls) as the first-class schedulable unit. We present SAGA, a distributed scheduler that implements this abstraction through three mechanisms: (1) Agent Execution Graphs that capture workflow structure to predict KV cache reuse across tool-call boundaries, achieving within 1.31x of B'el'ady's optimal offline policy; (2) session-affinity batching with work stealing that co-locates correlated requests while maintaining global load balance; and (3) Agent Fair Share, a task-completion-time fairness metric with provable bounded-deviation guarantees. On a 64-GPU cluster serving SWE-bench coding agents and WebArena browser tasks, SAGA reduces task completion time by 1.64x (geometric mean, p < 0.001) over vLLM v0.15.1 with prefix caching and affinity routing, while improving GPU memory utilization by 1.22x and achieving 99.2% SLO attainment under multi-tenant interference. These latency gains come at a quantified cost: approximately 30% lower peak throughput than throughput-optimal batch scheduling, a tradeoff appropriate for the latency-sensitive interactive deployments that dominate compound AI usage. Our results demonstrate that workflow-aware scheduling is essential for efficient compound AI serving.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 83 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

34 of 83 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Sherlock: Reliable and Efficient Agentic Workflow Execution1 Nov
  2. 2025Pie: A Programmable Serving System for Emerging LLM Applications28 Oct
  3. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  4. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents10 Apr
  5. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  6. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
16 more
  1. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  6. 2024AI Agents That Mattercited by 199
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  8. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  9. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  10. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  11. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  12. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  13. 2023Fairness in Serving Large Language Modelscited by 77
  14. 2019Themis: Fair and Efficient GPU Cluster Schedulingcited by 43
  15. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  16. 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
  2. 2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programscited by 1
  3. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  4. 2026Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Controlcited by 0
  5. 2026Towards Agentic Cloud Engineering: Graph and Loop Engineering with a Zero-Trust Agent Harnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems
  2. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective
  3. 2026SwarmX: Agentic Scheduling for Low-Latency Agentic Systems
  4. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  5. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
  6. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
  7. 2025Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering
  8. 2026Architectural Implications of Agentic AI Workflows