mlsanityData to 14 Oct 2026

ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding

arXiv 2604.13519 ↗First posted 15 Apr 2026cs.CLcited by 5

On the map: Retrieval-augmented generation → Generation and context integration

also in 2 more
Abstract (arXiv)

Tool calling has greatly expanded the practical utility of large language models (LLMs) by enabling them to interact with external applications. As LLM capabilities advance, effective tool use increasingly involves multi-step, multi-turn interactions to solve complex tasks. However, the resulting growth in tool interactions incurs substantial latency, posing a key challenge for real-time LLM serving. Through empirical analysis, we find that tool-calling traces are highly structured, conform to constrained schemas, and often exhibit recurring invocation patterns. Motivated by this, we propose ToolSpec, a schema-aware, retrieval-augmented speculative decoding method for accelerating tool calling. ToolSpec exploits predefined tool schemas to generate accurate drafts, using a finite-state machine to alternate between deterministic schema token filling and speculative generation for variable fields. In addition, ToolSpec retrieves similar historical tool invocations and reuses them as drafts to further improve efficiency. ToolSpec presents a plug-and-play solution that can be seamlessly integrated into existing LLM workflows. Experiments across multiple benchmarks demonstrate that ToolSpec achieves up to a 4.2x speedup, substantially outperforming existing training-free speculative decoding methods.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 27 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

20 of 27 references.

References · 20

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  9. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  10. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  11. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  12. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
8 more
  1. 2023An LLM Compiler for Parallel Function Callingcited by 179
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  3. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  4. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 47
  5. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 39
  6. 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Executioncited by 17
  7. 2025Divide-Then-Aggregate: An Efficient Tool Learning Method via Parallel Tool Invocationcited by 16
  8. 2025Optimizing Agentic Language Model Inference via Speculative Tool Callscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Callingcited by 1
  2. 2026SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decodingcited by 1
  3. 2026Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Toolscited by 0
  4. 2026Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computescited by 0
  5. 2026SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models
  2. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  3. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  4. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  5. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  6. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation
  7. 2025Speculative Decoding: Performance or Illusion?
  8. 2026Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets