mlsanityData to 7 Oct 2026

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

arXiv 2604.25899 ↗First posted 28 Apr 2026cs.MA cs.DC cs.SYcited by 6

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing structure that constrains agent behavior and exposes useful semantic predictability. Unlike traditional LLM serving, which operates under highly dynamic and uncertain conditions, this structured topology enables opportunities to reduce runtime uncertaintyunicodex2015yet existing systems fail to exploit it, treating agentic workloads as generic traffic and incurring significant inefficiencies. Our analysis of production traces from an agent-serving platform and an internal coding assistant reveals key bottlenecks, including low prefix cache hit rates, severe resource contention from long-context requests, and substantial queuing delays due to suboptimal scaling. To address these challenges, we propose Pythia, a multi-agent serving system that captures workflow semantics through a simple interface at the serving layer, unlocking new optimization opportunities and substantially improving throughput and job completion time over state-of-the-art baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 43 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

29 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System14 Feb
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  3. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  4. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug

Other references · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  8. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  9. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  10. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  11. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  12. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
12 more
  1. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  2. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  3. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  4. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  5. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  6. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  7. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  8. 2025Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analyticscited by 20
  9. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  10. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12
  11. 2024SYMPHONY: Improving Memory Management for LLM Inference Workloadscited by 5
  12. 2026Pancake: Hierarchical Memory System for Multi-Agent LLM Servingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
  2. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  3. 2026Architectural Implications of Agentic AI Workflowscited by 2
  4. 2026SwarmX: Agentic Scheduling for Low-Latency Agentic Systemscited by 0
  5. 2026TokenCast: Forecasting Token Consumption During LLM Agent Executioncited by 0
  6. 2026Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
  2. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
  3. 2026A Policy-Driven Runtime Layer for Agentic LLM Serving
  4. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
  5. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents
  6. 2026Substrate-Portable Execution for Production LLM Workflows
  7. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving
  8. 2025A Declarative Language for Building And Orchestrating LLM-Powered Agent Workflows