# UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

arXiv: https://arxiv.org/abs/2511.03334 · HTML: https://mlsanity.com/paper/2511.03334/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 30 of 52)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners](https://mlsanity.com/paper/2402.17723/) (2024)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [DDT: Decoupled Diffusion Transformer](https://mlsanity.com/paper/2504.05741/) (2025)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation](https://mlsanity.com/paper/2505.04512/) (2025)

## Built on by (most cited first, 30 of 38)

- [MOVA: Towards Scalable and Synchronized Video-Audio Generation](https://mlsanity.com/paper/2602.08794/) (2026)
- [T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.21094/) (2025)
- [JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion](https://mlsanity.com/paper/2601.22143/) (2026)
- [Native Audio-Visual Alignment for Generation](https://mlsanity.com/paper/2605.30073/) (2026)
- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)
- [PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence](https://mlsanity.com/paper/2512.13465/) (2025)
- [ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars](https://mlsanity.com/paper/2512.19546/) (2025)
- [MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation](https://mlsanity.com/paper/2512.03034/) (2025)
- [Improving Joint Audio-Video Generation with Cross-Modal Context Learning](https://mlsanity.com/paper/2603.18600/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation](https://mlsanity.com/paper/2605.28035/) (2026)
- [OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation](https://mlsanity.com/paper/2604.18326/) (2026)
- [Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling](https://mlsanity.com/paper/2604.23586/) (2026)
- [TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation](https://mlsanity.com/paper/2605.01809/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Benchmarking Single-Factor Physical Video-to-Audio Generation](https://mlsanity.com/paper/2605.30339/) (2026)
- [InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos](https://mlsanity.com/paper/2608.01157/) (2026)
- [Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation](https://mlsanity.com/paper/2609.36995/) (2026)
- [JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing](https://mlsanity.com/paper/2512.13677/) (2025)
- [Identity as Presence: Towards Appearance and Voice Personalized Joint Audio-Video Generation](https://mlsanity.com/paper/2603.17889/) (2026)
- [Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation](https://mlsanity.com/paper/2604.25819/) (2026)
- [AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation](https://mlsanity.com/paper/2606.30811/) (2026)
- [Cycle-World: Mitigating Error Accumulation in Long-term Video World Models via Reverse-Prediction Cycle Consistency](https://mlsanity.com/paper/2607.11836/) (2026)
- [Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention](https://mlsanity.com/paper/2608.15522/) (2026)
- [Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation](https://mlsanity.com/paper/2602.13585/) (2026)
- [Interactive Multi-Turn Retrieval for Health Videos](https://mlsanity.com/paper/2605.01409/) (2026)
- [OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars](https://mlsanity.com/paper/2607.23023/) (2026)
- [Vorch-Omni: Multi-Task Orchestration of Sight and Sound](https://mlsanity.com/paper/2608.05803/) (2026)
- [Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence](https://mlsanity.com/paper/2608.10720/) (2026)
- [DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution](https://mlsanity.com/paper/2608.31106/) (2026)

## Abstract (arXiv)

Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.