mlsanityData to 13 Oct 2026

PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

arXiv 2601.21124 ↗First posted 28 Jan 2026cs.SD cs.AI eess.AScited by 6

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 45 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

19 of 45 references.

References · 19

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2025Gemma 3 Technical Reportcited by 1,748
  4. 2021AST: Audio Spectrogram Transformercited by 639
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 229
  6. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 213
  7. 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 84
  8. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 46
  9. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 35
  10. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  11. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 16
  12. 2017Broadband DOA estimation using Convolutional neural networks trained with noise signalscited by 15
7 more
  1. 2024Can Large Language Models Understand Spatial Audio?cited by 14
  2. 2025SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editingcited by 6
  3. 2016Localization of Simultaneous Moving Sound Sources for Mobile Robot Using a Frequency-Domain Steered Beamformer Approachcited by 4
  4. 2024Ambisonics Encoding For Arbitrary Microphone Arrays Incorporating Residual Channels For Binaural Reproductioncited by 4
  5. 2024M-BEST-RQ: A Multi-Channel Speech Foundation Model for Smart Glassescited by 4
  6. 2025Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognitioncited by 3
  7. 2025SpeechCompass: Enhancing Mobile Captioning with Diarization and Directional Guidance via Multi-Microphone Localizationcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encodingcited by 6
  2. 2026OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agentscited by 1
  3. 2026SceneBind: Binding What and Where Across Vision, Audio and Languagecited by 0
  4. 2026SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Formercited by 0
  5. 2026SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrayscited by 0
  6. 2026FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
  2. 2025SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
  3. 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models
  4. 2026The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
  5. 2026JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
  6. 2025In-the-wild Audio Spatialization with Flexible Text-guided Localization
  7. 2025Spatial Audio Motion Understanding and Reasoning
  8. 2026The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization