# Sci-Phi: A Large Language Model Spatial Audio Descriptor

arXiv: https://arxiv.org/abs/2510.05542 · HTML: https://mlsanity.com/paper/2510.05542/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events](https://mlsanity.com/paper/2306.09126/) (2023)
- [BAT: Learning to Reason about Spatial Sounds with Large Language Models](https://mlsanity.com/paper/2402.01591/) (2024)
- [Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training](https://mlsanity.com/paper/2110.07124/) (2021)
- [Learning Spatially-Aware Language and Audio Embeddings](https://mlsanity.com/paper/2409.11369/) (2024)
- [STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events](https://mlsanity.com/paper/2206.01948/) (2022)
- [Can Large Language Models Understand Spatial Audio?](https://mlsanity.com/paper/2406.07914/) (2024)
- [Sound Event Localization based on Sound Intensity Vector Refined By DNN-Based Denoising and Source Separation](https://mlsanity.com/paper/2002.05994/) (2020)
- [SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing](https://mlsanity.com/paper/2507.16724/) (2025)
- [Echo-aware Adaptation of Sound Event Localization and Detection in Unknown Environments](https://mlsanity.com/paper/2202.09121/) (2022)
- [Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representation](https://mlsanity.com/paper/2309.15938/) (2023)
- [Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio](https://mlsanity.com/paper/2310.10922/) (2023)

## Built on by (most cited first, 5 of 5)

- [Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources](https://mlsanity.com/paper/2606.14141/) (2026)
- [OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents](https://mlsanity.com/paper/2609.23407/) (2026)
- [Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonics](https://mlsanity.com/paper/2609.23152/) (2026)
- [SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Former](https://mlsanity.com/paper/2609.34347/) (2026)
- [SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays](https://mlsanity.com/paper/2610.05610/) (2026)

## Abstract (arXiv)

Acoustic scene perception involves describing the type of sounds, their timing, their direction and distance, as well as their loudness and reverberation. While audio language models excel in sound recognition, single-channel input fundamentally limits spatial understanding. This work presents Sci-Phi, a spatial audio large language model with dual spatial and spectral encoders that estimates a complete parameter set for all sound sources and the surrounding environment. Learning from over 4,000 hours of synthetic first-order Ambisonics recordings including metadata, Sci-Phi enumerates and describes up to four directional sound sources in one pass, alongside non-directional background sounds and room characteristics. We evaluate the model with a permutation-invariant protocol and 15 metrics covering content, location, timing, loudness, and reverberation, and analyze its robustness across source counts, signal-to-noise ratios, reverberation levels, and challenging mixtures of acoustically, spatially, or temporally similar sources. Notably, Sci-Phi generalizes to real room impulse responses with only minor performance degradation. Overall, this work establishes the first audio LLM capable of full spatial-scene description, with strong potential for real-world deployment. Demo: https://sci-phi-audio.github.io/demo