# Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking

arXiv: https://arxiv.org/abs/2511.01299 · HTML: https://mlsanity.com/paper/2511.01299/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2025.

## Built on by (most cited first, 4 of 4)

- [Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2603.14636/) (2026)
- [Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models](https://mlsanity.com/paper/2605.20356/) (2026)
- [Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems](https://mlsanity.com/paper/2601.20230/) (2026)
- [Augmenting Dysarthric Speech Severity Assessment with MOS Supervision](https://mlsanity.com/paper/2606.18645/) (2026)

## Abstract (arXiv)

In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities of foundation models, towards more comprehensive understanding, more natural generation and more human-like interaction. Audio, as a modality rich in semantic, emotional, and contextual cues, plays a vital role in achieving naturalistic and embodied machine intelligence. This survey provides a comprehensive review of recent progress in integrating audio into LLMs, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. We analyze how LLMs are reshaping audio perception and reasoning, enabling systems to understand sound at a deeper semantic level, generate expressive audio outputs, and engage in human-like spoken interaction. Furthermore, we explore how the fusion of audio and visual modalities enhances situational awareness and cross-modal reasoning, pushing the boundaries of multimodal intelligence. This survey not only synthesizes existing research but also identifies critical challenges and future directions for building audio-native AGI systems capable of perceiving, understanding, and interacting through sound as naturally as humans do.