mlsanityData to 14 Oct 2026

AV-MaskEnhancer: Enhancing Video Representations through Audio-Visual Masked Autoencoder

arXiv 2309.08738 ↗First posted 15 Sep 2023cs.CV cs.MMcited by 11

Not on the map. Why?

Abstract (arXiv)

Learning high-quality video representation has shown significant applications in computer vision and remains challenging. Previous work based on mask autoencoders such as ImageMAE and VideoMAE has proven the effectiveness of learning representations in images and videos through reconstruction strategy in the visual modality. However, these models exhibit inherent limitations, particularly in scenarios where extracting features solely from the visual modality proves challenging, such as when dealing with low-resolution and blurry original videos. Based on this, we propose AV-MaskEnhancer for learning high-quality video representation by combining visual and audio information. Our approach addresses the challenge by demonstrating the complementary nature of audio and video features in cross-modality content. Moreover, our result of the video classification task on the UCF101 dataset outperforms the existing work and reaches the state-of-the-art, with a top-1 accuracy of 98.8% and a top-5 accuracy of 99.9%.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

24 of 27 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
12 more
  1. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  3. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  4. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  5. 2022Masked Autoencoders that Listencited by 273
  6. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  7. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  8. 2020In defence of metric learning for speaker recognitioncited by 126
  9. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  10. 2020Evolving Losses for Unsupervised Video Representation Learningcited by 74
  11. 2020Learning Spatiotemporal Features via Video and Text Pair Discriminationcited by 38
  12. 2020DASGIL: Domain Adaptation for Semantic and Geometric-aware Image-based Localizationcited by 28

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 5.

11 papers.

  1. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  2. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  3. 2023SAIC: Integration of Speech Anonymization and Identity Classificationcited by 7
  4. 2024Efflex: Efficient and Flexible Pipeline for Spatio-Temporal Trajectory Graph Modeling and Representation Learningcited by 5
  5. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  6. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4
  7. 2024VeTraSS: Vehicle Trajectory Similarity Search Through Graph Modeling and Representation Learningcited by 3
  8. 2024CrossGP: Cross-Day Glucose Prediction Excluding Physiological Informationcited by 2
  9. 2024GluMarker: A Novel Predictive Modeling of Glycemic Control Through Digital Biomarkerscited by 2
  10. 2024Toward Short-Term Glucose Prediction Solely Based on CGM Time Seriescited by 1
  11. 2025CPCLDETECTOR: Knowledge Enhancement and Alignment Selection for Chinese Patronizing and Condescending Language Detectioncited by 0