mlsanityData to 14 Oct 2026

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

arXiv 2412.15322 ↗First posted 19 Dec 2024cs.CV cs.LG cs.SDcited by 134

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio. In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code and demo are available at: https://hkchengrex.github.io/MMAudio

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). Builds on 60 of its 75 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

60 of 75 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  11. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
38 more
  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  2. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  3. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  4. 2017Self-Normalizing Neural Networkscited by 598
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  7. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  8. 2019Clotho: An Audio Captioning Datasetcited by 358
  9. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  10. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  11. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  12. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  13. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  14. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  15. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  16. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  17. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  18. 2021Taming Visually Guided Sound Generationcited by 136
  19. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  20. 2015Visually Indicated Soundscited by 128
  21. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  22. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  23. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 97
  24. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  25. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  26. 2020Generating Visually Aligned Sound from Videoscited by 67
  27. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  28. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  29. 2023Natural Language Supervision for General-Purpose Audio Representationscited by 61
  30. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  31. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  32. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  33. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  34. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  35. 2023FoleyGen: Visually-Guided Audio Generationcited by 40
  36. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  37. 2024Audio-Synchronized Visual Animationcited by 32
  38. 2024Text-to-Audio Generation Synchronized with Videoscited by 28

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 52; 2026: 82.

134 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  3. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  4. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 46
  5. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  6. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  7. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  8. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 31
  9. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 30
  10. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 27
  11. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  12. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
38 more
  1. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 24
  2. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generationcited by 21
  3. 2025The Curse of Conditions: Analyzing and Improving Optimal Transport for Conditional Flow-Based Generationcited by 17
  4. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  5. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 16
  6. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 13
  7. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 12
  8. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 12
  9. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 11
  10. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 11
  11. 2026MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generationcited by 10
  12. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  13. 2025Long-Video Audio Synthesis with Multi-Agent Collaborationcited by 8
  14. 2025FoleyBench: A Benchmark For Video-to-Audio Modelscited by 7
  15. 2025Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Editscited by 7
  16. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  17. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 7
  18. 2026Native Audio-Visual Alignment for Generationcited by 7
  19. 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generationcited by 6
  20. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  21. 2026CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generationcited by 6
  22. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  23. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  24. 2025AudioMoG: Guiding Audio Generation with Mixture-of-Guidancecited by 5
  25. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 5
  26. 2025InstructAudio: Unified speech and music generation with natural language instructioncited by 5
  27. 2026AVControl: Efficient Framework for Training Audio-Visual Controlscited by 5
  28. 2025DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videoscited by 4
  29. 2025DDL: A Large-Scale Dataset for Deepfake Detection and Localization in Diversified Real-World Scenarioscited by 4
  30. 2025Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Videocited by 4
  31. 2025Tortoise and Hare Guidance: Accelerating Diffusion Model Inference with Multirate Integrationcited by 4
  32. 2025Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removalcited by 4
  33. 2026AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfercited by 4
  34. 2026Woosh: A Sound Effects Foundation Modelcited by 4
  35. 2026Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worldscited by 4
  36. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  37. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  38. 2025MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separationcited by 3

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Time Scaling for Joint Audio-Video Generation
  2. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
  3. 2026Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning
  4. 2026Soundwich: Video Generation with Layered and Controllable Audio
  5. 2026Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
  6. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
  7. 2024Video-to-Audio Generation with Fine-grained Temporal Semantics
  8. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation