mlsanityData to 7 Oct 2026

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

arXiv 2212.09478 ↗First posted 19 Dec 2022cs.CVcited by 225

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.

Key facts. Cited by 225 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 46 of its 54 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

46 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 45

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
33 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  5. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  6. 2022Denoising Diffusion Restoration Modelscited by 951
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  8. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  9. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  10. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  11. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  12. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  13. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  14. 2022AudioGen: Textually Guided Audio Generationcited by 292
  15. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  16. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  17. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  18. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  19. 2017Deep Cross-Modal Audio-Visual Generationcited by 63
  20. 2021Video Background Music Generation with Controllable Music Transformercited by 55
  21. 2021Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessmentcited by 41
  22. 2021Domain-Aware Universal Style Transfercited by 40
  23. 2022Sound-Guided Semantic Video Generationcited by 35
  24. 20224D LUT: Learnable Context-Aware 4D Lookup Table for Image Enhancementcited by 34
  25. 2017CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generationcited by 31
  26. 2020Sound2Sight: Generating Visual Dynamics from Sound and Contextcited by 28
  27. 2022Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolutioncited by 25
  28. 2021Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformerscited by 19
  29. 2022TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolationcited by 14
  30. 2022Language-Guided Face Animation by Recurrent StyleGAN-based Generatorcited by 9
  31. 2021Learning Fine-Grained Motion Embedding for Landscape Animationcited by 7
  32. 2022Degradation-Guided Meta-Restoration Network for Blind Super-Resolutioncited by 7
  33. 2022AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 65; 2025: 71; 2026: 52.

225 papers.

Sort:

By area:

  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  3. 2023A Survey on Video Diffusion Modelscited by 203
  4. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  5. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  6. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  7. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  8. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  9. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  10. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  11. 2025Controllable Video Generation: A Surveycited by 93
  12. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
38 more
  1. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  2. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNetcited by 83
  3. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  4. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  5. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  6. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  7. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 63
  8. 2023Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generationcited by 60
  9. 2023AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Posecited by 60
  10. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  11. 2024Zero-Reference Low-Light Enhancement via Physical Quadruple Priorscited by 58
  12. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  13. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  14. 2023EDMP: Ensemble-of-costs-guided Diffusion for Motion Planningcited by 49
  15. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  16. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  17. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  18. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Imagescited by 42
  19. 2024Diffusion Facial Forgery Detectioncited by 41
  20. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39
  21. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  22. 2023Connecting Multi-modal Contrastive Representationscited by 36
  23. 2023Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Imagescited by 36
  24. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  25. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  26. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  27. 2023Generative-based Fusion Mechanism for Multi-Modal Trackingcited by 33
  28. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  29. 2024Audio-Synchronized Visual Animationcited by 32
  30. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  31. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  32. 2023Stable Diffusion is Unstablecited by 27
  33. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  34. 2025Multi-modal Time Series Analysis: A Tutorial and Surveycited by 24
  35. 2025TransAnimate: Taming Layer Diffusion to Generate RGBA Videocited by 24
  36. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  37. 2024AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoisingcited by 22
  38. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22

50 most cited of 225 All 225 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Training-Free Multimodal Guidance for Video to Audio Generation
  2. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
  3. 2023ArchiSound: Audio Generation with Diffusion
  4. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
  5. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
  6. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learning
  7. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion
  8. 2024PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models