# SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text

arXiv: https://arxiv.org/abs/2412.15220 · HTML: https://mlsanity.com/paper/2412.15220/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2024.

## Built on by (most cited first, 26 of 26)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [UniVerse-1: Unified Audio-Video Generation via Stitching of Experts](https://mlsanity.com/paper/2509.06155/) (2025)
- [JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation](https://mlsanity.com/paper/2602.19163/) (2026)
- [DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2602.12160/) (2026)
- [EchoWM: Open and Enterable Omnimodal World Models](https://mlsanity.com/paper/2608.23189/) (2026)
- [Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction](https://mlsanity.com/paper/2510.03117/) (2025)
- [PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.23994/) (2025)
- [Apollo: Unified Multi-Task Audio-Video Joint Generation](https://mlsanity.com/paper/2601.04151/) (2026)
- [MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning](https://mlsanity.com/paper/2601.01568/) (2026)
- [Flow Straight and Fast in Hilbert Space: Functional Rectified Flow](https://mlsanity.com/paper/2509.10384/) (2025)
- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)
- [MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation](https://mlsanity.com/paper/2512.03034/) (2025)
- [Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal](https://mlsanity.com/paper/2512.12875/) (2025)
- [Inference-Time Scaling for Joint Audio-Video Generation](https://mlsanity.com/paper/2606.03183/) (2026)
- [Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?](https://mlsanity.com/paper/2602.01623/) (2026)
- [Bridging Your Imagination with Audio-Video Generation via a Unified Director](https://mlsanity.com/paper/2512.23222/) (2025)
- [Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling](https://mlsanity.com/paper/2604.23586/) (2026)
- [Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer](https://mlsanity.com/paper/2608.15690/) (2026)
- [Controllable Audio-Visual Viewpoint Generation from 360deg Spatial Information](https://mlsanity.com/paper/2510.06060/) (2025)
- [Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence](https://mlsanity.com/paper/2608.10720/) (2026)
- [DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution](https://mlsanity.com/paper/2608.31106/) (2026)
- [Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy](https://mlsanity.com/paper/2609.34381/) (2026)
- [Soundwich: Video Generation with Layered and Controllable Audio](https://mlsanity.com/paper/2610.00691/) (2026)
- [Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation](https://mlsanity.com/paper/2610.05608/) (2026)
- [Generative World Simulation with DAO and JING](https://mlsanity.com/paper/2610.19726/) (2026)

## Abstract (arXiv)

Video and audio are closely correlated modalities that humans naturally perceive together. While recent advancements have enabled the generation of audio or video from text, producing both modalities simultaneously still typically relies on either a cascaded process or multi-modal contrastive encoders. These approaches, however, often lead to suboptimal results due to inherent information losses during inference and conditioning. In this paper, we introduce SyncFlow, a system that is capable of simultaneously generating temporally synchronized audio and video from text. The core of SyncFlow is the proposed dual-diffusion-transformer (d-DiT) architecture, which enables joint video and audio modelling with proper information fusion. To efficiently manage the computational cost of joint audio and video modelling, SyncFlow utilizes a multi-stage training strategy that separates video and audio learning before joint fine-tuning. Our empirical evaluations demonstrate that SyncFlow produces audio and video outputs that are more correlated than baseline methods with significantly enhanced audio quality and audio-visual correspondence. Moreover, we demonstrate strong zero-shot capabilities of SyncFlow, including zero-shot video-to-audio generation and adaptation to novel video resolutions without further training.