# AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks

arXiv: https://arxiv.org/abs/2506.23049 · HTML: https://mlsanity.com/paper/2506.23049/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities](https://mlsanity.com/paper/2410.11190/) (2024)
- [Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models](https://mlsanity.com/paper/2503.02318/) (2025)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents](https://mlsanity.com/paper/2305.13040/) (2023)
- [Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction](https://mlsanity.com/paper/2502.17239/) (2025)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models](https://mlsanity.com/paper/2503.00564/) (2025)
- [ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents](https://mlsanity.com/paper/2411.00927/) (2024)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)

## Built on by (most cited first, 5 of 5)

- [Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage](https://mlsanity.com/paper/2510.02044/) (2025)
- [From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench](https://mlsanity.com/paper/2604.15037/) (2026)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)
- [The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge](https://mlsanity.com/paper/2609.11724/) (2026)
- [Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents](https://mlsanity.com/paper/2609.32536/) (2026)

## Abstract (arXiv)

Despite advances in language and speech technologies, no open-source system enables full speech-to-speech, multi-turn dialogue with integrated tool use and agentic reasoning. We introduce AURA (Agent for Understanding, Reasoning, and Automated Tool Use), the first open-source, speech-native assistant capable of completing complex, goal-driven tasks through dynamic tool invocation and multi-turn conversation. AURA combines open-weight ASR, TTS, and LLMs in a cascaded pipeline and supports tools such as calendar booking, contact lookup, web search, and email. Its modular design allows easy integration of new tools using natural language prompts and action classes. On VoiceBench, AURA scores 92.75% on OpenBookQA-outperforming all open-weight systems and nearing GPT-4o-and 4.39 on AlpacaEval, competitive with other open-weight systems. Human evaluation shows 90% task success on complex, multi-turn speech tasks.