Native Audio-Visual Alignment for Generation
Not on the map. Why?
Abstract (arXiv)
Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
24 of 28 references.
References · 24
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
- 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
- 2026Seedance 2.0: Advancing Video Generation for World Complexitycited by 160
12 more
- 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
- 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
- 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 88
- 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
- 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
- 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
- 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
- 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
- 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
- 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 25
- 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 20
- 2024Video-to-Audio Generation with Fine-grained Temporal Semanticscited by 6
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layercited by 1
- 2026Vorch-Omni: Multi-Task Orchestration of Sight and Soundcited by 0
- 2026DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolutioncited by 0
- 2026TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Modelscited by 0
- 2026The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generationcited by 0
- 2026LongLive-Plug: Once-for-All Distillation for Video Generationcited by 0
- 2026SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generationcited by 0
- 2026SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation12 Oct 2026
- 2026LongLive-Plug: Once-for-All Distillation for Video Generation29 Sep 2026
- 2026The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation2 Sep 2026
- 2026TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models1 Sep 2026
- 2026DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution31 Aug 2026
- 2026Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer16 Aug 2026
- 2026Vorch-Omni: Multi-Task Orchestration of Sight and Sound6 Aug 2026