mlsanityData to 14 Oct 2026

Optimizing Agentic Language Model Inference via Speculative Tool Calls

arXiv 2512.15834 ↗First posted 17 Dec 2025cs.PL cs.AI cs.DCcited by 13

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 21 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

17 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Large Language Model-Based Agents for Software Engineering: A Survey4 Sep
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 15

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2023Tool Learning with Foundation Modelscited by 302
  9. 2024Tool Learning with Large Language Models: A Surveycited by 223
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  11. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  12. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
3 more
  1. 2024The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systemscited by 29
  2. 2024An LLM-Tool Compiler for Fused Parallel Function Callingcited by 15
  3. 2025Understanding and Improving Communication Performance in Multi-node LLM Inferencecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  2. 2026Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Usecited by 8
  3. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inferencecited by 6
  4. 2026ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decodingcited by 5
  5. 2026Leyline: KV Cache Directives for Agentic Inferencecited by 5
  6. 2026Multimodal Latent Reasoning via Predictive Embeddingscited by 4
  7. 2026IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agentscited by 4
  8. 2026AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Servingcited by 3
  9. 2026Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RLcited by 2
  10. 2026AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMscited by 1
  11. 2026Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Toolscited by 0
  12. 2026LEAP: Learning Efficient Action Proposals For LLM Agentscited by 0
1 more
  1. 2026ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  2. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and System
  3. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
  4. 2026AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
  5. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
  6. 2025Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offs
  7. 2026Stateful Inference for Low-Latency Multi-Agent Tool Calling
  8. 2026Harnessing LLMs as Agents: What Does It Cost?