mlsanityData to 14 Oct 2026

PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers

arXiv 2111.12710 ↗First posted 24 Nov 2021cs.CV cs.LGcited by 204

Not on the map. Why?

Abstract (arXiv)

This paper explores a better prediction target for BERT pre-training of vision transformers. We observe that current prediction targets disagree with human perception judgment.This contradiction motivates us to learn a perceptual prediction target. We argue that perceptually similar images should stay close to each other in the prediction target space. We surprisingly find one simple yet effective idea: enforcing perceptual similarity during the dVAE training. Moreover, we adopt a self-supervised transformer model for deep feature extraction and show that it works well for calculating perceptual similarity.We demonstrate that such learned visual tokens indeed exhibit better semantic meanings, and help pre-training achieve superior transfer performance in various downstream tasks. For example, we achieve 84.5% Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive method BEiT by +1.3% under the same pre-training epochs. Our approach also gets significant improvement on object detection and segmentation on COCO and semantic segmentation on ADE20K. Equipped with a larger backbone ViT-H, we achieve the state-of-the-art ImageNet accuracy (88.3%) among methods using only ImageNet-1K data.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). Builds on 64 of its 91 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

64 of 91 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
38 more
  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2017Mask R-CNNcited by 7,277
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  13. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  14. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  15. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  16. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  17. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  18. 2017Non-local Neural Networkscited by 2,293
  19. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  20. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  21. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  22. 2017The Kinetics Human Action Video Datasetcited by 1,909
  23. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  24. 2016Adversarial examples in the physical worldcited by 1,769
  25. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  26. 2018Group Normalizationcited by 1,524
  27. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  28. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  29. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  30. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  31. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  32. 2016Deep Networks with Stochastic Depthcited by 979
  33. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  34. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  35. 2016Pixel Recurrent Neural Networkscited by 816
  36. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  37. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  38. 2020Self-supervised Learning: Generative or Contrastivecited by 694

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 78; 2023: 61; 2024: 34; 2025: 25; 2026: 4.

204 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  3. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  4. 2022DeiT III: Revenge of the ViTcited by 419
  5. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  6. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  7. 2023A Cookbook of Self-Supervised Learningcited by 250
  8. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  9. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  10. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  11. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  12. 2022MIC: Masked Image Consistency for Context-Enhanced Domain Adaptationcited by 178
38 more
  1. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  4. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  5. 2022VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decodercited by 116
  6. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  7. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  8. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  9. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  10. 2022Three things everyone should know about Vision Transformerscited by 90
  11. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  12. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  13. 2022What do Vision Transformers Learn? A Visual Explorationcited by 76
  14. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 71
  15. 2023Diffusion Models as Masked Autoencoderscited by 71
  16. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  17. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  18. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  19. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  20. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  21. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  22. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  23. 2022Mugs: A Multi-Granular Self-Supervised Learning Frameworkcited by 54
  24. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  25. 2022SdAE: Self-distillated Masked Autoencodercited by 52
  26. 2024Restructuring Vector Quantization with the Rotation Trickcited by 51
  27. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  28. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  29. 2022Dissecting Self-Supervised Learning Methods for Surgical Computer Visioncited by 48
  30. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  31. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 47
  32. 2023Regularized Vector Quantization for Tokenized Image Synthesiscited by 46
  33. 2022Masked Image Modeling with Denoising Contrastcited by 45
  34. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  35. 2022Masked Event Modeling: Self-Supervised Pretraining for Event Camerascited by 41
  36. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  37. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  38. 2023GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Trainingcited by 38

50 most cited of 204 All 204 (JSON)