mlsanityData to 7 Oct 2026

HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling

arXiv 2605.16637 ↗First posted 15 May 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Agentic LLM applications increasingly execute user requests as multi-step workflows involving planning, tool use, branching, refinement, and synthesis. In such settings, users experience the end-to-end latency of an entire workflow, not the latency of any single LLM call. In this paper, we study how to schedule online agentic workflows across heterogeneous prefill-decode disaggregated LLM serving clusters to efficiently meet workflow-level latency objectives. The problem is challenging because workflow dependencies are revealed incrementally at runtime, calls have heterogeneous prompts, outputs, and KV-cache requirements, and the prefill and decode stages impose different compute, memory, and transfer constraints across heterogeneous GPUs. To solve this problem, we present HexAGenT, a workflow-aware scheduler for a heterogeneous prefill-decode inference service. HexAGenT models each request as an online-revealed DAG, maintains a running estimate of the workflow's standalone completion horizon, prioritizes ready calls by projected risk of missing that horizon, and jointly selects prefill placement, decode placement, and local queue priority while accounting for KV-cache capacity and cross-stage transfer latency. Across representative agentic workloads and heterogeneous A100/H100/H200 clusters, HexAGenT reduces the SLO scale required for timely workflow completion by an average of 20.1% at 95% attainment and 33.0% at 99% attainment, with maximum reductions of 45.0% and 80.5%, respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 52 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

39 of 52 references.

References · 39

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
27 more
  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  2. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  3. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  4. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  6. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  7. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  8. 2024Queue management for slo-oriented large language model servingcited by 36
  9. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  10. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  11. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  12. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  13. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  14. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  15. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  16. 2022OSDP: Optimal Sharded Data Parallel for Distributed Deep Learningcited by 12
  17. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  18. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  19. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  20. 2025Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUscited by 6
  21. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  22. 2026Efficient Multi-round LLM Inference over Disaggregated Servingcited by 6
  23. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  24. 2025HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQLcited by 3
  25. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  26. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  27. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
  2. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
  3. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
  4. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
  5. 2026Orla: A Library for Serving LLM-Based Multi-Agent Systems
  6. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  7. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving
  8. 2026Substrate-Portable Execution for Production LLM Workflows