mlsanityData to 7 Oct 2026

Contrastive Audio-Visual Masked Autoencoder

arXiv 2210.07839 ↗First posted 2 Oct 2022cs.MM cs.CV cs.SDcited by 142

Not on the map. Why?

Abstract (arXiv)

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 2 Oct 2022.

Builds on

51 of 60 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
38 more
  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  2. 2021AST: Audio Spectrogram Transformercited by 634
  3. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  4. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  5. 2017Look, Listen and Learncited by 452
  6. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  8. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  9. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  12. 2022Masked Autoencoders that Listencited by 271
  13. 2017Objects that Soundcited by 254
  14. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  15. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  16. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  17. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  18. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  19. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  20. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  21. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  22. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  23. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  24. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  25. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  26. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  27. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  28. 2021Robust Audio-Visual Instance Discriminationcited by 80
  29. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  30. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  31. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  32. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  33. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  34. 2020On Compositions of Transformations in Contrastive Self-Supervised Learningcited by 34
  35. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 31
  36. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27
  37. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  38. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 35; 2024: 44; 2025: 36; 2026: 24.

142 papers.

Sort:

By area:

  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  2. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  3. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  4. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  5. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  6. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  7. 2023Audio-Visual LLM for Video Understandingcited by 76
  8. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  9. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  10. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  11. 2022MAViL: Masked Audio-Video Learnerscited by 56
  12. 2022Audiovisual Masked Autoencoderscited by 47
38 more
  1. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  2. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  3. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  4. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2023Learning to Embed Time Series Patches Independentlycited by 34
  7. 2024Foundation Models for Music: A Surveycited by 34
  8. 2023Audio-Visual Class-Incremental Learningcited by 33
  9. 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
  10. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 32
  11. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  12. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  13. 2024Masked Image Modeling: A Surveycited by 28
  14. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  15. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  16. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  17. 2023Extending Multi-modal Contrastive Representationscited by 21
  18. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  19. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  20. 2024Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognitioncited by 18
  21. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  22. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  23. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  24. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 17
  25. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  26. 2023Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspectivecited by 14
  27. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
  28. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  29. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  30. 2023AV-MaskEnhancer: Enhancing Video Representations through Audio-Visual Masked Autoencodercited by 11
  31. 2024Revealing Vision-Language Integration in the Brain with Multimodal Networkscited by 11
  32. 2024EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learningcited by 10
  33. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  34. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  35. 2025Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noisescited by 9
  36. 2025Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metricscited by 9
  37. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  38. 2023SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extractioncited by 8

50 most cited of 142 All 142 (JSON)