mlsanityData to 14 Oct 2026

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

arXiv 2608.31106 ↗First posted 31 Aug 2026cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 72 of its 87 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

72 of 87 references.

References · 72

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2025DINOv3cited by 1,414
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  6. 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  8. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  9. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  10. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  12. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
38 more
  1. 2022MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessmentcited by 522
  2. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2021BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignmentcited by 342
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  6. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  7. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  8. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 252
  9. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  10. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  11. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  12. 2022VRT: A Video Restoration Transformercited by 220
  13. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  14. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
  15. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  16. 2021Investigating Tradeoffs in Real-World Video Super-Resolutioncited by 134
  17. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  18. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  19. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 132
  20. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  21. 2026Qwen3-ASR Technical Reportcited by 118
  22. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  23. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  24. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  25. 2024Video Diffusion Alignment via Reward Gradientscited by 107
  26. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 97
  27. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 88
  28. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  29. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  30. 2024VEnhancer: Generative Space-Time Enhancement for Video Generationcited by 64
  31. 2025SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restorationcited by 62
  32. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 61
  33. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
  34. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  35. 2025FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolutioncited by 43
  36. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  37. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 41
  38. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 40

50 most cited of 72 All 72 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model
  2. 2026Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
  3. 2026Diffusion Models for Joint Audio-Video Generation
  4. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
  5. 2026Soundwich: Video Generation with Layered and Controllable Audio
  6. 2025DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
  7. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion
  8. 2026AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation