mlsanityData to 14 Oct 2026

MAViL: Masked Audio-Video Learners

arXiv 2212.08071 ↗First posted 15 Dec 2022cs.CV cs.MM cs.SDcited by 59

Not on the map. Why?

Abstract (arXiv)

We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 83 of its 108 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

83 of 108 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2018SlowFast Networks for Video Recognitioncited by 1,867
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  10. 2019When Does Label Smoothing Help?cited by 988
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2021Multiscale Vision Transformerscited by 897
  13. 2019Relational Knowledge Distillationcited by 889
  14. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  15. 2019Contrastive Representation Distillationcited by 748
  16. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  17. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  18. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  19. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  20. 2021AST: Audio Spectrogram Transformercited by 643
  21. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  22. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  23. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  24. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  25. 2017Look, Listen and Learncited by 458
  26. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  27. 2019On the Efficacy of Knowledge Distillationcited by 381
  28. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  29. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  30. 2019Clotho: An Audio Captioning Datasetcited by 358
  31. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  32. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  33. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  34. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  35. 2022Masked Autoencoders that Listencited by 273
  36. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  37. 2017Objects that Soundcited by 256
  38. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 22; 2025: 12; 2026: 8.

59 papers.

Sort:

By area:

  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 169
  2. 2024OmniSat: Self-Supervised Modality Fusion for Earth Observationcited by 63
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  5. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
  6. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 51
  7. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  8. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 42
  9. 2024Foundation Models for Music: A Surveycited by 37
  10. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  11. 2024Masked Image Modeling: A Surveycited by 29
  12. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
38 more
  1. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  2. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  3. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  4. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  5. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  6. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 18
  7. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 16
  8. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  9. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  10. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  11. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  12. 2024Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learningcited by 10
  13. 2023Masked Autoencoders are Scalable Learners of Cellular Morphologycited by 9
  14. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  15. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  16. 2024Unsupervised Audio-Visual Segmentation with Modality Alignmentcited by 7
  17. 2024AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioningcited by 6
  18. 2023FLAP: Fast Language-Audio Pre-trainingcited by 5
  19. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  20. 2025VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injectioncited by 5
  21. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  22. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  23. 2024From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modelingcited by 4
  24. 2024Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation modelscited by 4
  25. 2025A Comprehensive Survey on Generative AI for Video-to-Music Generationcited by 4
  26. 2024MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognitioncited by 3
  27. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  28. 2023Diffusion Models as Masked Audio-Video Learnerscited by 2
  29. 2024Sequential Contrastive Audio-Visual Learningcited by 2
  30. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  31. 2025Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videoscited by 2
  32. 2026MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learningcited by 2
  33. 2023AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codescited by 1
  34. 2023Audio classification with Dilated Convolution with Learnable Spacingscited by 1
  35. 2025Structured-Noise Masked Modeling for Video, Audio and Beyondcited by 1
  36. 2026Audio-Visual Continual Test-Time Adaptation without Forgettingcited by 1
  37. 2026Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flowscited by 1
  38. 2026Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detectioncited by 1

50 most cited of 59 All 59 (JSON)