mlsanityData to 7 Oct 2026

A Policy-Driven Runtime Layer for Agentic LLM Serving

arXiv 2605.27744 ↗First posted 26 May 2026cs.AIcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, role, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents. A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other. We argue this seam is best addressed by an architectural change rather than point fixes: insert a third tier, an agent runtime layer, between the framework and the engine, exposing four primitives (observe, score, predict, act) into which any agent-aware policy plugs, with agent identity as the shared coordinate. We map nine concrete policies onto the layer and validate the abstraction in depth on the one with the largest immediate serving-cost lever: KV caching across sessions, instantiated as CacheScout, which learns the per-workload agent transition matrix online and uses it for survival-based eviction and between-step prefetch. Preliminary results on five real multi-agent workloads show +13 to +37 pp cache hit-rate lift, 12% to 29% lower mean TTFT, and 6% to 14% higher throughput over an unmodified serving stack.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 38 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

26 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  2. 2025Sherlock: Reliable and Efficient Agentic Workflow Execution1 Nov
  3. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems14 Oct
  4. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  5. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC30 Jun
  6. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  7. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving5 Nov

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
7 more
  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  2. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  3. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  4. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  6. 2024Fast Inference for Augmented Large Language Modelscited by 10
  7. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  2. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0
  3. 2026Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
  2. 2026Agentao: A Policy-Governed Runtime Harness for Embeddable Tool-Using LLM Agents
  3. 2026A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
  4. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  5. 2026From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution
  6. 2026AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programs
  7. 2026Agentic AI Workload Characteristics
  8. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing