mlsanityData to 14 Oct 2026

Diffusion Models as Masked Audio-Video Learners

arXiv 2310.03937 ↗First posted 5 Oct 2023cs.SD cs.CV cs.MMcited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Over the past several years, the synchronization between audio and visual signals has been leveraged to learn richer audio-visual representations. Aided by the large availability of unlabeled videos, many unsupervised training frameworks have demonstrated impressive results in various downstream audio and video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. MAViL couples contrastive learning with masked autoencoding to jointly reconstruct audio spectrograms and video frames by fusing information from both modalities. In this paper, we study the potential synergy between diffusion models and MAViL, seeking to derive mutual benefits from these two frameworks. The incorporation of diffusion into MAViL, combined with various training efficiency methodologies that include the utilization of a masking ratio curriculum and adaptive batch sizing, results in a notable 32% reduction in pre-training Floating-Point Operations (FLOPS) and an 18% decrease in pre-training wall clock time. Crucially, this enhanced efficiency does not compromise the model's performance in downstream audio-classification tasks when compared to MAViL's performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 41 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

35 of 41 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
23 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  8. 2016Deep Networks with Stochastic Depthcited by 979
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  13. 2017Look, Listen and Learncited by 458
  14. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  15. 2022Masked Autoencoders that Listencited by 273
  16. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  17. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  18. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  19. 2023Diffusion Models as Masked Autoencoderscited by 71
  20. 2022MAViL: Masked Audio-Video Learnerscited by 59
  21. 2022CVNets: High Performance Library for Computer Visioncited by 16
  22. 2022ConvMixer: Feature Interactive Convolution with Curriculum Learning for Small Footprint and Noisy Far-field Keyword Spottingcited by 11
  23. 2023On the Efficacy of Multi-scale Data Samplers for Vision Applicationscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Foundation Models for Music: A Surveycited by 37
  2. 2024Unsupervised Audio-Visual Segmentation with Modality Alignmentcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
  2. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
  3. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
  4. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
  5. 2023Masked Diffusion Models Are Fast Distribution Learners
  6. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
  7. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representations
  8. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization