mlsanityData to 14 Oct 2026

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models

arXiv 2608.01881 ↗First posted 3 Aug 2026cs.MM cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

16 of 26 references.

References · 16

  1. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  2. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  4. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  5. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  6. 2022ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wildcited by 121
  7. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  8. 2021AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenariocited by 60
  9. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluencycited by 37
  10. 2025MiDashengLM: Efficient Audio Understanding with General Audio Captionscited by 33
  11. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Usecited by 9
  12. 2025AudioGenie-Reasoner: A Training-Free Multi-Agent Framework for Coarse-to-Fine Audio Deep Reasoningcited by 6
4 more
  1. 2026AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoningcited by 6
  2. 2026AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoningcited by 3
  3. 2026Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoningcited by 3
  4. 2026EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
  2. 2026Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
  3. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception
  4. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
  5. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  6. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
  7. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
  8. 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing