# Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding

arXiv: https://arxiv.org/abs/2606.10738 · HTML: https://mlsanity.com/paper/2606.10738/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 33 of its 75 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2408.00765/) (2024)
- [BAT: Learning to Reason about Spatial Sounds with Large Language Models](https://mlsanity.com/paper/2402.01591/) (2024)
- [SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition](https://mlsanity.com/paper/2511.21471/) (2025)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [OmniAudio: Generating Spatial Audio from 360-Degree Video](https://mlsanity.com/paper/2504.14906/) (2025)
- [STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events](https://mlsanity.com/paper/2206.01948/) (2022)
- [Can Large Language Models Understand Spatial Audio?](https://mlsanity.com/paper/2406.07914/) (2024)
- [OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models](https://mlsanity.com/paper/2509.26140/) (2025)
- [STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence](https://mlsanity.com/paper/2510.24693/) (2025)
- [SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models](https://mlsanity.com/paper/2511.06606/) (2025)
- [SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing](https://mlsanity.com/paper/2507.16724/) (2025)
- [ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting](https://mlsanity.com/paper/2504.20630/) (2025)
- [The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models](https://mlsanity.com/paper/2601.02954/) (2026)
- [PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs](https://mlsanity.com/paper/2601.21124/) (2026)
- [JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments](https://mlsanity.com/paper/2602.18527/) (2026)
- [Spatial Audio Processing with Large Language Model on Wearable Devices](https://mlsanity.com/paper/2504.08907/) (2025)
- [Towards Spatial Audio Understanding via Question Answering](https://mlsanity.com/paper/2507.09195/) (2025)

## Built on by (most cited first, 7 of 7)

- [OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents](https://mlsanity.com/paper/2609.23407/) (2026)
- [Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models](https://mlsanity.com/paper/2608.09435/) (2026)
- [OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents](https://mlsanity.com/paper/2609.32569/) (2026)
- [AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models](https://mlsanity.com/paper/2610.06949/) (2026)
- [SkillFormer: Skill-Decomposed Adaptation for Audio Language Models](https://mlsanity.com/paper/2610.07533/) (2026)
- [SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding](https://mlsanity.com/paper/2610.10624/) (2026)
- [SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning](https://mlsanity.com/paper/2610.17131/) (2026)

## Abstract (arXiv)

Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at https://github.com/dieKarotte/Spatial-Omni.