mlsanityData to 7 Oct 2026

Context Autoencoder for Self-Supervised Representation Learning

arXiv 2202.03026 ↗First posted 7 Feb 2022cs.CVcited by 297

Not on the map. Why?

Abstract (arXiv)

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches. The network is an encoder-regressor-decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://github.com/Atten4Vis/CAE.

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Builds on 60 of its 114 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

60 of 114 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  14. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  15. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  16. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  18. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  19. 2016Colorful Image Colorizationcited by 1,113
  20. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  21. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  22. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  23. 2016Deep Networks with Stochastic Depthcited by 973
  24. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  25. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  26. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  27. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  28. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  29. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  30. 2017Large Batch Training of Convolutional Networkscited by 529
  31. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  32. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  33. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  34. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  35. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  36. 2020Whitening for Self-Supervised Representation Learningcited by 278
  37. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  38. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 73; 2023: 98; 2024: 68; 2025: 38; 2026: 20.

297 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  7. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  8. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  9. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  10. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  11. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 180
  12. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
38 more
  1. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  2. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  3. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  4. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  5. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  6. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  7. 2022Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignmentcited by 96
  8. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  9. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 92
  10. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  11. 2022E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Contextcited by 76
  12. 2022Masked Image Modeling Advances 3D Medical Image Analysiscited by 71
  13. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
  14. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 70
  15. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  16. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  17. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  18. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  19. 2022Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoderscited by 57
  20. 2023Hard Patches Mining for Masked Image Modelingcited by 56
  21. 2022SdAE: Self-distillated Masked Autoencodercited by 52
  22. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  23. 2023PLIP: Language-Image Pre-training for Person Representation Learningcited by 49
  24. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 46
  25. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  26. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  27. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  28. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  29. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  30. 2023GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Trainingcited by 38
  31. 2022A Unified View of Masked Image Modelingcited by 36
  32. 2024LW-DETR: A Transformer Replacement to YOLO for Real-Time Detectioncited by 36
  33. 2022CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNetcited by 35
  34. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  35. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  36. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
  37. 2022Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformerscited by 33
  38. 2023HAP: Structure-Aware Masked Image Modeling for Human-Centric Perceptioncited by 33

50 most cited of 297 All 297 (JSON)