mlsanityData to 7 Oct 2026

MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation

arXiv 2410.02130 ↗First posted 3 Oct 2024cs.SD cs.CV eess.AScited by 9

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, textttMDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9% alignment accuracy, using 172× fewer parameters, 371% less memory, and offering 36× faster inference than the current 860M-parameter state-of-the-art model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5× fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 47 of its 55 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

47 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MDT-A2G: Exploring Masked Diffusion Transformers for Co-Speech Gesture Generation6 Aug
  2. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generation24 May
  3. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models29 Jun
  4. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 43

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
31 more
  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  2. 2022Masked Autoencoders that Listencited by 271
  3. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  5. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  6. 2021Taming Visually Guided Sound Generationcited by 136
  7. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  8. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  9. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  10. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  11. 2022Localizing Visual Sounds the Easy Waycited by 81
  12. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  13. 2020Generating Visually Aligned Sound from Videoscited by 67
  14. 2019Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolutioncited by 57
  15. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  16. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  17. 2020Modality Shifting Attention Network for Multi-modal Video Question Answeringcited by 38
  18. 2024FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editingcited by 31
  19. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  20. 2023CDPMSR: Conditional Diffusion Probabilistic Models for Single Image Super-Resolutioncited by 21
  21. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  22. 2023ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolutioncited by 18
  23. 2024FlexiFilm: Long Video Generation with Flexible Conditionscited by 13
  24. 2024Cross-view Masked Diffusion Transformers for Person Image Synthesiscited by 12
  25. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  26. 2023Learning from Multi-Perception Features for Real-Word Image Super-resolutioncited by 9
  27. 2022LAD: A Hybrid Deep Learning System for Benign Paroxysmal Positional Vertigo Disorders Diagnosticcited by 8
  28. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  29. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  30. 2021Self-supervised Learning with Local Attention-Aware Featurecited by 5
  31. 2023Implicit Steganography Beyond the Constraints of Modalitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 2.

9 papers.

  1. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10
  2. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  3. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  4. 2025E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customizationcited by 2
  5. 2026A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformerscited by 2
  6. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 1
  7. 2025SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videoscited by 0
  8. 2025Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generationcited by 0
  9. 2026AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splattingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
  2. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  3. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  4. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
  5. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
  6. 2023EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis
  7. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
  8. 2026Diffusion Models for Joint Audio-Video Generation