mlsanityData to 14 Oct 2026

HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

arXiv 2607.22578 ↗First posted 7 Jun 2026cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration, HeraSys eliminates cross-workflow computational redundancy via structural node merging and reuse. Furthermore, HeraSys introduces a load-aware joint scheduling policy that dynamically manages execution order by evaluating both inter- and intra-query priorities. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2.17× and increases serving throughput by up to 1.85× under strict latency guarantees.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 41 references that are in the map. First posted on arXiv 7 Jun 2026.

Builds on

25 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
  3. 2023ChatDev: Communicative Agents for Software Development16 Jul

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
10 more
  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  3. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  5. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  6. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  7. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
  8. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  9. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  10. 2025FlowMesh: A Service Fabric for Composable LLM Workflowscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
  2. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
  3. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
  4. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
  5. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
  6. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  7. 2025Niyama : Breaking the Silos of LLM Inference Serving
  8. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents