mlsanityData to 14 Oct 2026

Stateful Inference for Low-Latency Multi-Agent Tool Calling

arXiv 2605.26289 ↗First posted 25 May 2026cs.LGcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the O(nt) per-turn cost of conventional serving into an O(Deltat) delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is 2.1× faster per turn on a 6-turn agentic workflow and 4.2× on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 5 of its 7 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

5 of 7 references.

References · 5

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  5. 2026Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformerscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Pathcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  2. 2026Akashic: A Low-Overhead LLM Inference Service with MemAttention
  3. 2025MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  5. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
  6. 2026SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding
  7. 2026Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
  8. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference