mlsanityData to 14 Oct 2026

iBOT: Image BERT Pre-Training with Online Tokenizer

arXiv 2111.07832 ↗First posted 15 Nov 2021cs.CVcited by 903

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics. The online tokenizer is jointly learnable with the MIM objective and dispenses with a multi-stage training pipeline where the tokenizer needs to be pre-trained beforehand. We show the prominence of iBOT by achieving an 82.3% linear probing accuracy and an 87.8% fine-tuning accuracy evaluated on ImageNet-1K. Beyond the state-of-the-art image classification results, we underline emerging local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, eg., object detection, instance segmentation, and semantic segmentation.

Key facts. Cited by 903 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 50 of its 56 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

50 of 56 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2017Mask R-CNNcited by 7,277
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  14. 2019Natural Adversarial Examplescited by 1,408
  15. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  16. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  17. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  18. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  19. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  20. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  21. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  22. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  23. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  24. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  25. 2021Intriguing Properties of Vision Transformerscited by 446
  26. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  27. 2020SCAN: Learning to Classify Images without Labelscited by 352
  28. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  29. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  30. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  31. 2016Discrete Variational Autoencoderscited by 125
  32. 2021Self-Supervised Learning with Swin Transformerscited by 122
  33. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  34. 2021SiT: Self-supervised vIsion Transformercited by 99
  35. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  36. 2018Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learningcited by 51
  37. 2021Self-Supervised Visual Representations Learning by Contrastive Mask Predictioncited by 31
  38. 2021Self-Supervised Classification Networkcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 131; 2023: 195; 2024: 186; 2025: 191; 2026: 197.

903 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  4. 2023Vision Transformers Need Registerscited by 871
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  10. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  11. 2022DeiT III: Revenge of the ViTcited by 419
  12. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
38 more
  1. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  2. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  3. 2023RoMa: Robust Dense Feature Matchingcited by 302
  4. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  5. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  6. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  7. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  8. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 262
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  11. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  12. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  13. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  14. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  15. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  16. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  17. 2022Masked World Models for Visual Controlcited by 170
  18. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  19. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  20. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  21. 2024Multimodal Whole Slide Foundation Model for Pathologycited by 162
  22. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  23. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  24. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  25. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  26. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  27. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  28. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  29. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  30. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  31. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 132
  32. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  33. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  34. 20234M: Massively Multimodal Masked Modelingcited by 120
  35. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  36. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  37. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  38. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117

50 most cited of 903 All 903 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
  2. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
  3. 2021Data Efficient Masked Language Modeling for Vision and Language
  4. 2024Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning
  5. 2025FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression
  6. 2023What Makes for Good Visual Tokenizers for Large Language Models?
  7. 2023Masking meets Supervision: A Strong Learning Alliance
  8. 2022Token Dropping for Efficient BERT Pretraining