# StereoSync: Spatially-Aware Stereo Audio Generation from Video

arXiv: https://arxiv.org/abs/2510.05828 · HTML: https://mlsanity.com/paper/2510.05828/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [I Hear Your True Colors: Image Guided Audio Generation](https://mlsanity.com/paper/2211.03089/) (2022)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [Generating Visually Aligned Sound from Videos](https://mlsanity.com/paper/2008.00820/) (2020)
- [2.5D Visual Sound](https://mlsanity.com/paper/1812.04204/) (2018)
- [Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation](https://mlsanity.com/paper/2007.09902/) (2020)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [Matching Anything by Segmenting Anything](https://mlsanity.com/paper/2406.04221/) (2024)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Video Depth without Video Models](https://mlsanity.com/paper/2411.19189/) (2024)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis](https://mlsanity.com/paper/2310.15247/) (2023)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning](https://mlsanity.com/paper/2002.10981/) (2020)
- [T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis](https://mlsanity.com/paper/2401.09294/) (2024)

## Abstract (arXiv)

Although audio generation has been widely studied over recent years, video-aligned audio generation still remains a relatively unexplored frontier. To address this gap, we introduce StereoSync, a novel and efficient model designed to generate audio that is both temporally synchronized with a reference video and spatially aligned with its visual context. Moreover, StereoSync also achieves efficiency by leveraging pretrained foundation models, reducing the need for extensive training while maintaining high-quality synthesis. Unlike existing methods that primarily focus on temporal synchronization, StereoSync introduces a significant advancement by incorporating spatial awareness into video-aligned audio generation. Indeed, given an input video, our approach extracts spatial cues from depth maps and bounding boxes, using them as cross-attention conditioning in a diffusion-based audio generation model. Such an approach allows StereoSync to go beyond simple synchronization, producing stereo audio that dynamically adapts to the spatial structure and movement of a video scene. We evaluate StereoSync on Walking The Maps, a curated dataset comprising videos from video games that feature animated characters walking through diverse environments. Experimental results demonstrate the ability of StereoSync to achieve both temporal and spatial alignment, advancing the state of the art in video-to-audio generation and resulting in a significantly more immersive and realistic audio experience.