mlsanityData to 7 Oct 2026

AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks

arXiv 2506.23049 ↗First posted 29 Jun 2025cs.AI cs.CL cs.SDcited by 5

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

Despite advances in language and speech technologies, no open-source system enables full speech-to-speech, multi-turn dialogue with integrated tool use and agentic reasoning. We introduce AURA (Agent for Understanding, Reasoning, and Automated Tool Use), the first open-source, speech-native assistant capable of completing complex, goal-driven tasks through dynamic tool invocation and multi-turn conversation. AURA combines open-weight ASR, TTS, and LLMs in a cascaded pipeline and supports tools such as calendar booking, contact lookup, web search, and email. Its modular design allows easy integration of new tools using natural language prompts and action classes. On VoiceBench, AURA scores 92.75% on OpenBookQA-outperforming all open-weight systems and nearing GPT-4o-and 4.39 on AlpacaEval, competitive with other open-weight systems. Human evaluation shows 90% task success on complex, multi-turn speech tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 33 references that are in the map. First posted on arXiv 29 Jun 2025.

Builds on

23 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models4 Mar
  2. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model13 Jan
  3. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming29 Aug

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2025Kimi-Audio Technical Reportcited by 210
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
8 more
  1. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  2. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  3. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  4. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  5. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Modelscited by 34
  6. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 14
  7. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  8. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 12
  2. 2026From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Benchcited by 3
  3. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agentscited by 0
  4. 2026The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challengecited by 0
  5. 2026Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  2. 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing
  3. 2026AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery
  4. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
  5. 2026AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models
  6. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
  7. 2026Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
  8. 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants