mlsanityData to 19 Oct 2026

MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

arXiv 2610.11156 ↗First posted 8 Oct 2026cs.SD eess.AScited by 0

On the map: Speech and audio language models → Audio language models and spoken dialogue

Abstract (arXiv)

Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models. Builds on 49 of its 66 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

49 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning25 Aug
  2. 2026Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources12 Jun
  3. 2026MOSS-Audio Technical Report1 Jun
  4. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music13 Apr
  5. 2026ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding25 Mar
  6. 2025SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models10 Nov
  7. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence28 Oct
  8. 2025MiDashengLM: Efficient Audio Understanding with General Audio Captions6 Aug
  9. 2025SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing22 Jul
  10. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models10 Jul
  11. 2024Qwen2-Audio Technical Report15 Jul
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models14 Nov

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,672
  2. 2025Qwen3 Technical Reportcited by 9,258
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,450
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,058
  5. 2025Qwen2.5-Omni Technical Reportcited by 835
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 831
  7. 2025Qwen3-Omni Technical Reportcited by 578
  8. 2026Gemma 4 Technical Reportcited by 298
  9. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  10. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 247
  11. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 175
  12. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 149
25 more
  1. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 97
  2. 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 90
  3. 2025MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligencecited by 81
  4. 2024Scaling up masked audio encoder learning for general audio classificationcited by 53
  5. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 45
  6. 2021EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environmentscited by 41
  7. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 41
  8. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 39
  9. 2020Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019cited by 32
  10. 2020ACCDOA: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization and Detectioncited by 30
  11. 2020An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detectioncited by 29
  12. 2021Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Trainingcited by 29
  13. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 22
  14. 2021SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detectioncited by 17
  15. 2019First Order Ambisonics Domain Spatial Augmentation for DNN-based Direction of Arrival Estimationcited by 13
  16. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Modelscited by 12
  17. 2024RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localizationcited by 9
  18. 2024PSELDNets: Pre-trained Neural Networks on a Large-scale Synthetic Dataset for Sound Event Localization and Detectioncited by 9
  19. 2026Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encodingcited by 8
  20. 2022A Track-Wise Ensemble Event Independent Network for Polyphonic Sound Event Localization and Detectioncited by 7
  21. 2026PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMscited by 7
  22. 2025Stereo Sound Event Localization and Detection with Onscreen/offscreen Classificationcited by 5
  23. 2025Spatial-CLAP: Learning Spatially-Aware audio--text Embeddings for Multi-Source Conditionscited by 5
  24. 2025MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotationscited by 4
  25. 2025An Experimental Study on Joint Modeling for Sound Event Localization and Detection with Source Distance Estimationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
  2. 2026Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs
  3. 2024Can Large Language Models Understand Spatial Audio?
  4. 2025Spatial Audio Motion Understanding and Reasoning
  5. 2026SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays
  6. 2025Sci-Phi: A Large Language Model Spatial Audio Descriptor
  7. 2026SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Former
  8. 2026SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding