mlsanityData to 7 Oct 2026

Audiovisual Masked Autoencoders

arXiv 2212.05922 ↗First posted 9 Dec 2022cs.CV cs.SDcited by 47

Not on the map. Why?

Abstract (arXiv)

Can we leverage the audiovisual information already present in video to improve self-supervised representation learning? To answer this question, we study various pretraining architectures and objectives within the masked autoencoding framework, motivated by the success of similar methods in natural language and image understanding. We show that we can achieve significant improvements on audiovisual downstream classification tasks, surpassing the state-of-the-art on VGGSound and AudioSet. Furthermore, we can leverage our audiovisual pretraining scheme for multiple unimodal downstream tasks using a single audiovisual pretrained model. We additionally demonstrate the transferability of our representations, achieving state-of-the-art audiovisual results on Epic Kitchens without pretraining specifically for this dataset.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 74 of its 91 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

74 of 91 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021ViViT: A Video Vision Transformercited by 1,610
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  13. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  14. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  16. 2016Deep Networks with Stochastic Depthcited by 973
  17. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  18. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  19. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  20. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  21. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  22. 2021AST: Audio Spectrogram Transformercited by 634
  23. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  24. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  25. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  26. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  27. 2020Rescaling Egocentric Visioncited by 529
  28. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  29. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  30. 2017Look, Listen and Learncited by 452
  31. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  32. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  33. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  34. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  35. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  36. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  37. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  38. 2020Self-Supervised MultiModal Versatile Networkscited by 276

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 23; 2025: 11; 2026: 2.

47 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  2. 2023Audio-Visual LLM for Video Understandingcited by 76
  3. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  4. 2022MAViL: Masked Audio-Video Learnerscited by 56
  5. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  6. 2024Masked Image Modeling: A Surveycited by 28
  7. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  8. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  9. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 21
  10. 2024Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognitioncited by 18
  11. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generationcited by 17
  12. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 17
35 more
  1. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  2. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
  3. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  4. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  5. 2023Learning from One Continuous Video Streamcited by 11
  6. 2024Planted: a dataset for planted forest identification from multi-satellite time seriescited by 9
  7. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  8. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  9. 2024Unsupervised Audio-Visual Segmentation with Modality Alignmentcited by 7
  10. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  11. 2024AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioningcited by 6
  12. 2025Decoupled Audio-Visual Dataset Distillationcited by 6
  13. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  14. 2024Answering Diverse Questions via Text Attached with Key Audio-Visual Cluescited by 4
  15. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  16. 2023On Robustness in Multimodal Learningcited by 3
  17. 2023Cascaded Cross-Modal Transformer for Request and Complaint Detectioncited by 3
  18. 2024CBM: Curriculum by Maskingcited by 3
  19. 2024Pre-training with Synthetic Patterns for Audiocited by 3
  20. 2024Cascaded Cross-Modal Transformer for Audio-Textual Classificationcited by 2
  21. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  22. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  23. 2025Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videoscited by 2
  24. 2024DETECLAP: Enhancing Audio-Visual Representation Learning with Object Informationcited by 1
  25. 2026Audio-Visual Continual Test-Time Adaptation without Forgettingcited by 1
  26. 2024Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Datacited by 0
  27. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignmentcited by 0
  28. 2024KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encodercited by 0
  29. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  30. 2025SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verificationcited by 0
  31. 2025Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Groundingcited by 0
  32. 2025Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognitioncited by 0
  33. 2025Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voicecited by 0
  34. 2025Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Videocited by 0
  35. 2026Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learningcited by 0