Data to 7 Oct 2026

Autellix: An Efficient Serving Engine for LLM Agents as General Programs

arXiv 2502.13965 ↗First posted 19 Feb 2025cs.LG cs.AI cs.DCcited by 57

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more

Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs, which scale LLM calls and output tokens to help AI agents reason, explore, and solve complex tasks. However, existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization. Our analysis reveals that programs submitted to LLM serving engines experience long cumulative wait times, primarily due to head-of-line blocking at both the individual LLM request and the program. To address this, we introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies. Autellix intercepts LLM calls submitted by programs, enriching schedulers with program-level context. We propose two scheduling algorithms-for single-threaded and distributed programs-that preempt and prioritize LLM calls based on their programs' previously completed calls. Our evaluation demonstrates that across diverse LLMs and agentic workloads, Autellix improves throughput of programs by 4-15x at the same latency compared to state-of-the-art systems, such as vLLM.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 62 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

22 of 62 references.

References · 22

2020Language Models are Few-Shot Learnerscited by 32,199
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302

10 more

2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
2023Fast Distributed Inference Serving for Large Language Modelscited by 143
2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 43.

57 papers.

Sort:

By area:

2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 17
2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13

38 more

2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
2025Sherlock: Reliable and Efficient Agentic Workflow Executioncited by 10
2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 10
2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 6
2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 5
2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systemscited by 5
2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
2025Astraea: A State-Aware Scheduling Engine for LLM-Powered Agentscited by 3
2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 3
2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflowscited by 2
2026MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systemscited by 2
2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 2
2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
2026Software-Defined Agentic Servingcited by 1
2026Nalar: An agent serving frameworkcited by 1
2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programscited by 1
2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Servingcited by 1
2026Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Servedcited by 0
2026AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agentcited by 0
2026Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregationcited by 0
2026Beyond Prediction: Tail-Aware Scheduling for LLM Inferencecited by 0
2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Servingcited by 0
2026Scalable LLM Agent Tool Access in the Cloudcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way