mlsanityData to 7 Oct 2026

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

arXiv 2202.03382 ↗First posted 7 Feb 2022cs.CV cs.AI cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial [MASK] tokens, where some patches are randomly selected and replaced with plausible alternatives sampled from the BEiT output distribution. Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not. The generator and the enhancer are simultaneously trained and synergistically updated. After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks. CIM is a general and flexible visual pre-training framework that is suitable for various network architectures. For the first time, CIM demonstrates that both ViT and CNN can learn rich visual representations using a unified, non-Siamese framework. Experimental results show that our approach achieves compelling results in vision benchmarks, such as ImageNet classification and ADE20K semantic segmentation.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

54 of 73 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  14. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  17. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  18. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  19. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,352
  20. 2017Random Erasing Data Augmentationcited by 1,235
  21. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  22. 2018Self-Attention with Relative Position Representationscited by 1,112
  23. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  24. 2016Deep Networks with Stochastic Depthcited by 973
  25. 2020Pre-Trained Image Processing Transformercited by 891
  26. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  27. 2021Going deeper with Image Transformerscited by 759
  28. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  29. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  30. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  31. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  32. 2019Fixing the train-test resolution discrepancycited by 209
  33. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  34. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  35. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  36. 2016Discrete Variational Autoencoderscited by 124
  37. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  38. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 25; 2023: 24; 2024: 15; 2025: 4.

68 papers.

Sort:

By area:

  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  2. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  3. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  4. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  5. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  6. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 92
  7. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  8. 2022Self-Training: A Surveycited by 85
  9. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 70
  10. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  11. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  12. 2022Masked Image Modeling with Denoising Contrastcited by 45
38 more
  1. 2022On Data Scaling in Masked Image Modelingcited by 45
  2. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  3. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  4. 2022Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detectioncited by 33
  5. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
  6. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  7. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  8. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  9. 2022Efficient Self-supervised Vision Pretraining with Local Masked Reconstructioncited by 27
  10. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  11. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  12. 2022HiViT: Hierarchical Vision Transformer Meets Masked Image Modelingcited by 23
  13. 2023ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Processcited by 19
  14. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18
  15. 2023DeepMIM: Deep Supervision for Masked Image Modelingcited by 16
  16. 2022Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformerscited by 14
  17. 2023Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learningcited by 14
  18. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  19. 2022Masked Autoencoders are Robust Data Augmentorscited by 13
  20. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  21. 2023Correlational Image Modeling for Self-Supervised Visual Pre-Trainingcited by 11
  22. 2023Pre-training with Random Orthogonal Projection Image Modelingcited by 11
  23. 2023MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignmentscited by 9
  24. 2023CtxMIM: Context-Enhanced Masked Image Modeling for Remote Sensing Image Understandingcited by 9
  25. 2024MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learningcited by 8
  26. 2024A Survey of the Self Supervised Learning Mechanisms for Vision Transformerscited by 8
  27. 2025UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Trainingcited by 8
  28. 2022Pre-trained Token-replaced Detection Model as Few-shot Learnercited by 7
  29. 2023Beyond Pretrained Features: Noisy Image Modeling Provides Adversarial Defensecited by 7
  30. 2024Self-Supervised Alignment Learning for Medical Image Segmentationcited by 7
  31. 2022Representation Learning by Detecting Incorrect Location Embeddingscited by 5
  32. 2023Asymmetric Patch Sampling for Contrastive Learningcited by 5
  33. 2023Masked Image Residual Learning for Scaling Deeper Vision Transformerscited by 5
  34. 2022Self-Supervised Learning based on Heat Equationcited by 4
  35. 2024Frequency-Guided Masking for Enhanced Vision Self-Supervised Learningcited by 4
  36. 2023Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Trainingcited by 3
  37. 2024Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learningcited by 3
  38. 2024Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyondcited by 3

50 most cited of 68 All 68 (JSON)