mlsanityData to 7 Oct 2026

ConvMAE: Masked Convolution Meets Masked Autoencoders

arXiv 2205.03892 ↗First posted 8 May 2022cs.CVcited by 105

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained ConvMAE models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at https://github.com/Alpha-VL/ConvMAE.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 67 of its 75 references that are in the map. First posted on arXiv 8 May 2022.

Builds on

67 of 75 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  13. 2017The Kinetics Human Action Video Datasetcited by 1,894
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  15. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  16. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  17. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  18. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  20. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  21. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  22. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  23. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  24. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  25. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  26. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  27. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  28. 2021Transformer in Transformercited by 632
  29. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  30. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  31. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  32. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  33. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  34. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  35. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  36. 2021Early Convolutions Help Transformers See Bettercited by 395
  37. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  38. 2017Temporal Relational Reasoning in Videoscited by 324

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 45; 2024: 20; 2025: 9; 2026: 8.

105 papers.

Sort:

By area:

  1. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  2. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 302
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  5. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  6. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  7. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 129
  8. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  9. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  10. 2023Lightweight, Pre-trained Transformers for Remote Sensing Timeseriescited by 102
  11. 2024Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observationcited by 97
  12. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 92
38 more
  1. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  2. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  3. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  4. 2022GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Cloudscited by 57
  5. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  6. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  7. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  8. 2023Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-trainingcited by 52
  9. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  10. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  11. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  12. 2022A Unified View of Masked Image Modelingcited by 36
  13. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  14. 2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
  15. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  16. 2023Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matterscited by 27
  17. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  18. 2023Unsupervised Anomaly Detection in Medical Images Using Masked Diffusion Modelcited by 25
  19. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  20. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  21. 2024Masked Angle-Aware Autoencoder for Remote Sensing Imagescited by 22
  22. 2023Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoderscited by 17
  23. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  24. 2022IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectorscited by 12
  25. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  26. 2024UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretrainingcited by 12
  27. 2025A versatile foundation model for cine cardiac magnetic resonance image analysis taskscited by 12
  28. 2022Recurrent Bilinear Optimization for Binary Neural Networkscited by 11
  29. 2023CMAE-V: Contrastive Masked Autoencoders for Video Action Recognitioncited by 11
  30. 2024CooPre: Cooperative Pretraining for V2X Cooperative Perceptioncited by 11
  31. 2022Towards Efficient Use of Multi-Scale Features in Transformer-Based Object Detectorscited by 10
  32. 2023FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentationcited by 10
  33. 2023Understanding Masked Autoencoders From a Local Contrastive Perspectivecited by 10
  34. 2024How Well Can Vision Language Models See Image Details?cited by 10
  35. 2024Zero-Shot Multi-Object Scene Completioncited by 9
  36. 2022Towards Sustainable Self-supervised Learningcited by 8
  37. 2022i-MAE: Are Latent Representations in Masked Autoencoders Linearly Separable?cited by 8
  38. 2023CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasetscited by 8

50 most cited of 105 All 105 (JSON)