# Asynchronous LLM Function Calling

arXiv: https://arxiv.org/abs/2412.07017 · HTML: https://mlsanity.com/paper/2412.07017/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models](https://mlsanity.com/paper/2305.18323/) (2023)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [Fast State Restoration in LLM Serving with HCache](https://mlsanity.com/paper/2410.05004/) (2024)
- [TypeFly: Flying Drones with Large Language Model](https://mlsanity.com/paper/2312.14950/) (2023)
- [Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution](https://mlsanity.com/paper/2406.00059/) (2024)
- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching](https://mlsanity.com/paper/2406.06799/) (2024)

## Built on by (most cited first, 18 of 18)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](https://mlsanity.com/paper/2511.02230/) (2025)
- [Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows](https://mlsanity.com/paper/2602.14849/) (2026)
- [Accelerating Earth Science Discovery via Multi-Agent LLM Systems](https://mlsanity.com/paper/2503.05854/) (2025)
- [SHIELDA: Structured Handling of Exceptions in LLM-Driven Agentic Workflows](https://mlsanity.com/paper/2508.07935/) (2025)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [Asynchronous Reasoning: Training-Free Interactive Thinking LLMs](https://mlsanity.com/paper/2512.10931/) (2025)
- [IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents](https://mlsanity.com/paper/2605.22154/) (2026)
- [ATTS: Asynchronous Test-Time Scaling via Conformal Prediction](https://mlsanity.com/paper/2509.15148/) (2025)
- [SimpleTool: Parallel Decoding for Real-Time LLM Function Calling](https://mlsanity.com/paper/2603.00030/) (2026)
- [Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling](https://mlsanity.com/paper/2605.13360/) (2026)
- [Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs](https://mlsanity.com/paper/2605.15077/) (2026)
- [Never Stop Thinking: Continuous-Time Language Agents](https://mlsanity.com/paper/2609.17416/) (2026)
- [ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling](https://mlsanity.com/paper/2507.08877/) (2025)
- [From Specification to Service: Accelerating API-First Development Using Multi-Agent Systems](https://mlsanity.com/paper/2510.19274/) (2025)
- [Serve Programs, Not Prompts](https://mlsanity.com/paper/2510.25412/) (2025)
- [AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito](https://mlsanity.com/paper/2601.18381/) (2026)
- [Second Thought: Reasoning in Parallel as LLM Agents Act and Observe](https://mlsanity.com/paper/2608.13667/) (2026)

## Abstract (arXiv)

Large language models (LLMs) use function calls to interface with external tools and data source. However, the current approach to LLM function calling is inherently synchronous, where each call blocks LLM inference, limiting LLM operation and concurrent function execution. In this work, we propose AsyncLM, a system for asynchronous LLM function calling. AsyncLM improves LLM's operational efficiency by enabling LLMs to generate and execute function calls concurrently. Instead of waiting for each call's completion, AsyncLM introduces an interrupt mechanism to asynchronously notify the LLM in-flight when function calls return. We design an in-context protocol for function calls and interrupts, provide fine-tuning strategy to adapt LLMs to the interrupt semantics, and implement these mechanisms efficiently on LLM inference process. We demonstrate that AsyncLM can reduce end-to-end task completion latency from 1.6x-5.4x compared to synchronous function calling on a set of benchmark tasks in the Berkeley function calling leaderboard (BFCL). Furthermore, we discuss how interrupt mechanisms can be extended to enable novel human-LLM or LLM-LLM interactions.