# Optimizing Agentic Language Model Inference via Speculative Tool Calls

arXiv: https://arxiv.org/abs/2512.15834 · HTML: https://mlsanity.com/paper/2512.15834/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning](https://mlsanity.com/paper/2410.12952/) (2024)
- [The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems](https://mlsanity.com/paper/2409.20002/) (2024)
- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [Understanding and Improving Communication Performance in Multi-node LLM Inference](https://mlsanity.com/paper/2511.09557/) (2025)

## Built on by (most cited first, 13 of 13)

- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use](https://mlsanity.com/paper/2602.11541/) (2026)
- [SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference](https://mlsanity.com/paper/2607.03333/) (2026)
- [ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding](https://mlsanity.com/paper/2604.13519/) (2026)
- [Leyline: KV Cache Directives for Agentic Inference](https://mlsanity.com/paper/2606.01065/) (2026)
- [Multimodal Latent Reasoning via Predictive Embeddings](https://mlsanity.com/paper/2604.08065/) (2026)
- [IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents](https://mlsanity.com/paper/2605.22154/) (2026)
- [AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving](https://mlsanity.com/paper/2608.00881/) (2026)
- [Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL](https://mlsanity.com/paper/2607.25816/) (2026)
- [AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs](https://mlsanity.com/paper/2608.26004/) (2026)
- [Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools](https://mlsanity.com/paper/2606.02483/) (2026)
- [LEAP: Learning Efficient Action Proposals For LLM Agents](https://mlsanity.com/paper/2610.02670/) (2026)
- [ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation](https://mlsanity.com/paper/2610.15246/) (2026)

## Abstract (arXiv)

Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations.