# BinauralVAE: Spatial Audio Reconstruction For World Models

arXiv: https://arxiv.org/abs/2609.06837 · HTML: https://mlsanity.com/paper/2609.06837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 6 Sep 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 22 of 22)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL](https://mlsanity.com/paper/2602.13977/) (2026)
- [Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video](https://mlsanity.com/paper/2404.09833/) (2024)
- [Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing](https://mlsanity.com/paper/2604.10708/) (2026)
- [Binaural Speech Enhancement Using Deep Complex Convolutional Transformer Networks](https://mlsanity.com/paper/2403.05393/) (2024)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [Binaural Sound Event Localization and Detection based on HRTF Cues for Humanoid Robots](https://mlsanity.com/paper/2507.20530/) (2025)
- [Semantic Audio-Visual Navigation in Continuous Environments](https://mlsanity.com/paper/2603.19660/) (2026)
- [AudioWorldSim: Realistic Binaural Audio Datasets For World Models](https://mlsanity.com/paper/2608.21075/) (2026)
- [Audio-Visual World Models: Learning Physically Grounded Multisensory Dynamics](https://mlsanity.com/paper/2512.00883/) (2025)

## Abstract (arXiv)

Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.