# Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models

arXiv: https://arxiv.org/abs/2608.01881 · HTML: https://mlsanity.com/paper/2608.01881/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency](https://mlsanity.com/paper/2604.04847/) (2026)
- [MiDashengLM: Efficient Audio Understanding with General Audio Captions](https://mlsanity.com/paper/2508.03983/) (2025)
- [Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use](https://mlsanity.com/paper/2604.22821/) (2026)
- [AudioGenie-Reasoner: A Training-Free Multi-Agent Framework for Coarse-to-Fine Audio Deep Reasoning](https://mlsanity.com/paper/2509.16971/) (2025)
- [AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoning](https://mlsanity.com/paper/2602.13685/) (2026)
- [AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoning](https://mlsanity.com/paper/2602.10439/) (2026)
- [Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning](https://mlsanity.com/paper/2602.11909/) (2026)
- [EChO-Agent: Evidence Chain Orchestration Agent for Audio Reasoning](https://mlsanity.com/paper/2606.15141/) (2026)

## Abstract (arXiv)

Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.