# Asynchronous Tool Usage for Real-Time Agents

arXiv: https://arxiv.org/abs/2410.21620 · HTML: https://mlsanity.com/paper/2410.21620/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 35 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 9 of 9)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration](https://mlsanity.com/paper/2310.02170/) (2023)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [Embodied LLM Agents Learn to Cooperate in Organized Teams](https://mlsanity.com/paper/2403.12482/) (2024)
- [Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration](https://mlsanity.com/paper/2405.14314/) (2024)

## Built on by (most cited first, 13 of 13)

- [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](https://mlsanity.com/paper/2511.02230/) (2025)
- [Large Language Models Miss the Multi-Agent Mark](https://mlsanity.com/paper/2505.21298/) (2025)
- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios](https://mlsanity.com/paper/2605.27995/) (2026)
- [Evaluating Speech-to-Text x LLM x Text-to-Speech Combinations for AI Interview Systems](https://mlsanity.com/paper/2507.16835/) (2025)
- [ATTS: Asynchronous Test-Time Scaling via Conformal Prediction](https://mlsanity.com/paper/2509.15148/) (2025)
- [AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning](https://mlsanity.com/paper/2510.16156/) (2025)
- [Engagement Process: Rethinking the Temporal Interface of Action and Observation](https://mlsanity.com/paper/2605.11484/) (2026)
- [Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs](https://mlsanity.com/paper/2605.15077/) (2026)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)
- [DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution](https://mlsanity.com/paper/2603.19248/) (2026)
- [Continuous Interaction Diffusion: A Diffusion-Native Runtime for Asynchronous Tool-Augmented Reasoning](https://mlsanity.com/paper/2608.10438/) (2026)
- [LLMs are General Asynchronous Agents](https://mlsanity.com/paper/2609.35427/) (2026)

## Abstract (arXiv)

While frontier large language models (LLMs) are capable tool-using agents, current AI systems still operate in a strict turn-based fashion, oblivious to passage of time. This synchronous design forces user queries and tool-use to occur sequentially, preventing the systems from multitasking and reducing interactivity. To address this limitation, we introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting, integrated with automatic speech recognition and text-to-speech. Drawing inspiration from the concepts originally developed for real-time operating systems, this work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.