# Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy

arXiv: https://arxiv.org/abs/2511.21579 · HTML: https://mlsanity.com/paper/2511.21579/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2025.

## Built on by (most cited first, 21 of 21)

- [MOVA: Towards Scalable and Synchronized Video-Audio Generation](https://mlsanity.com/paper/2602.08794/) (2026)
- [JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation](https://mlsanity.com/paper/2602.19163/) (2026)
- [T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.21094/) (2025)
- [MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data](https://mlsanity.com/paper/2606.02753/) (2026)
- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)
- [CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation](https://mlsanity.com/paper/2606.09639/) (2026)
- [PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence](https://mlsanity.com/paper/2512.13465/) (2025)
- [MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation](https://mlsanity.com/paper/2512.03034/) (2025)
- [Improving Joint Audio-Video Generation with Cross-Modal Context Learning](https://mlsanity.com/paper/2603.18600/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation](https://mlsanity.com/paper/2605.28035/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation](https://mlsanity.com/paper/2605.17488/) (2026)
- [Cycle-World: Mitigating Error Accumulation in Long-term Video World Models via Reverse-Prediction Cycle Consistency](https://mlsanity.com/paper/2607.11836/) (2026)
- [Vibe AIGC: A New Paradigm for Content Generation via Agentic Orchestration](https://mlsanity.com/paper/2602.04575/) (2026)
- [Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation](https://mlsanity.com/paper/2606.02441/) (2026)
- [TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models](https://mlsanity.com/paper/2609.01277/) (2026)
- [The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation](https://mlsanity.com/paper/2609.02367/) (2026)
- [All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation](https://mlsanity.com/paper/2609.27901/) (2026)
- [Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation](https://mlsanity.com/paper/2609.38748/) (2026)
- [Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training](https://mlsanity.com/paper/2610.05416/) (2026)

## Abstract (arXiv)

The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.