# WorldSonus: Bringing Sound to Worlds

arXiv: https://arxiv.org/abs/2610.08760 · HTML: https://mlsanity.com/paper/2610.08760/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 29 of 29)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [LongCat-Video Technical Report](https://mlsanity.com/paper/2510.22200/) (2025)
- [Infinite Worlds with Versatile Interactions](https://mlsanity.com/paper/2607.07534/) (2026)
- [HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation](https://mlsanity.com/paper/2508.16930/) (2025)
- [AudioX: A Unified Framework for Anything-to-Audio Generation](https://mlsanity.com/paper/2503.10522/) (2025)
- [T5Gemma 2: Seeing, Reading, and Understanding Longer](https://mlsanity.com/paper/2512.14856/) (2025)
- [EchoWM: Open and Enterable Omnimodal World Models](https://mlsanity.com/paper/2608.23189/) (2026)
- [Omni2Sound: Towards Unified Video-Text-to-Audio Generation](https://mlsanity.com/paper/2601.02731/) (2026)
- [PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation](https://mlsanity.com/paper/2511.18833/) (2025)
- [SoundReactor: Frame-level Online Video-to-Audio Generation](https://mlsanity.com/paper/2510.02110/) (2025)
- [StereoFoley: Object-Aware Stereo Audio Generation from Video](https://mlsanity.com/paper/2509.18272/) (2025)
- [AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer](https://mlsanity.com/paper/2603.15597/) (2026)
- [Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation](https://mlsanity.com/paper/2607.27372/) (2026)
- [Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds](https://mlsanity.com/paper/2608.23383/) (2026)
- [Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer](https://mlsanity.com/paper/2605.30940/) (2026)
- [Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory](https://mlsanity.com/paper/2607.26818/) (2026)
- [Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation](https://mlsanity.com/paper/2608.04902/) (2026)

## Abstract (arXiv)

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/