# Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

arXiv: https://arxiv.org/abs/2605.30940 · HTML: https://mlsanity.com/paper/2605.30940/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 50 of its 59 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization](https://mlsanity.com/paper/2404.09956/) (2024)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation](https://mlsanity.com/paper/2311.18775/) (2023)
- [Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching](https://mlsanity.com/paper/2406.00320/) (2024)
- [Temporally Aligned Audio for Video with Autoregression](https://mlsanity.com/paper/2409.13689/) (2024)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis](https://mlsanity.com/paper/2205.14807/) (2022)
- [FoleyGen: Visually-Guided Audio Generation](https://mlsanity.com/paper/2309.10537/) (2023)
- [ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing](https://mlsanity.com/paper/2506.21448/) (2025)
- [GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks](https://mlsanity.com/paper/2409.13832/) (2024)
- [Video-to-Audio Generation with Hidden Alignment](https://mlsanity.com/paper/2407.07464/) (2024)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video](https://mlsanity.com/paper/2111.10882/) (2021)

## Built on by (most cited first, 1 of 1)

- [WorldSonus: Bringing Sound to Worlds](https://mlsanity.com/paper/2610.08760/) (2026)

## Abstract (arXiv)

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.