mlsanityData to 7 Oct 2026

Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

arXiv 2609.18849 ↗First posted 16 Sep 2026cs.DC cs.AI cs.OScited by 0

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

An agentic request spends substantial wall-clock time waiting for tools, and its KV cache holds GPU memory the whole time. Serving systems decide whether that cache stays, leaves, or comes back by guessing how long the tool will run, from the tool's name, its history, a duration declared before the call, or the engine's own occupancy. We show that no estimate fixed before a call starts can know its duration, and such estimates may not even rank the calls. Meanwhile, the running tool already holds the answer, but the agent stack together with the tool silences it. We propose that tool calls report their progress explicitly while they run, and we measure what that takes. A census of four public agent corpora finds a readable signal in most tool time once it is revealed, in two strengths: a fraction of the work remaining, or an accurate signal that the end is near. A harness recovers it without changing what the agent sees, at no measurable cost to the agent's benchmark score. At the points where a KV cache decision is made, the reported progress is between several times and an order of magnitude more accurate than the best published predictors, and it stays accurate when the environment changes. Plugged into a production engine through a few small hints, it cuts the p90 time to first token (TTFT) after a tool call by 20.7% (HBM only) and 20.8% (HBM + DRAM) against LRU, close to an oracle. A serving system should not guess what its tools can tell it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 57 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

24 of 57 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems15 Aug
  2. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale30 Jul
  3. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Serving29 Jun
  4. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving1 Jun
  5. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System14 Feb
  6. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  7. 2025Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering19 Oct
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 15

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  5. 2020Optimal Robustness-Consistency Trade-offs for Learning-Augmented Online Algorithmscited by 64
  6. 2024The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systemscited by 29
  7. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  8. 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 16
  9. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  10. 2025Locality-aware Fair Scheduling in LLM Servingcited by 15
  11. 2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
  12. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
3 more
  1. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
  2. 2025Non-Clairvoyant Scheduling with Progress Barscited by 4
  3. 2026Preserving Admission Responsibility in Multi-Tenant Large Language Model Prefix Cachescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic AI Workload Characteristics
  2. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
  3. 2026ChurnBench: A Drift-Aware Benchmark Demonstrating That Refresh Scheduling, Not Cache Age, Governs Staleness in Agentic AI
  4. 2026Push Your Agent: Measuring and Enforcing Quantitative Goal Persistence in Long-Horizon LLM Agents
  5. 2026TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents
  6. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
  7. 2026The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation
  8. 2026Leyline: KV Cache Directives for Agentic Inference