mlsanityData to 7 Oct 2026

Emerging Properties in Self-Supervised Vision Transformers

arXiv 2104.14294 ↗First posted 29 Apr 2021cs.CVcited by 6,893

Not on the map. Why?

Cited by 615 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

Key facts. Cited by 6,893 papers in the map (data through 7 Oct 2026). Builds on 68 of its 91 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

68 of 91 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2016Colorful Image Colorizationcited by 1,113
  13. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  14. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  15. 2020Designing Network Design Spacescited by 954
  16. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  17. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  18. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  20. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  21. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  22. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 497
  23. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  24. 2020Meta Pseudo Labelscited by 372
  25. 2020Exploring Self-attention for Image Recognitioncited by 313
  26. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  27. 2020Whitening for Self-Supervised Representation Learningcited by 278
  28. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  29. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  30. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  31. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  32. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  33. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  34. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  35. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 189
  36. 2019Learning with Average Precision: Training Image Retrieval with a Listwise Losscited by 184
  37. 2015Particular object retrieval with integral max-pooling of CNN activationscited by 160
  38. 2018Large scale distributed neural network training through online distillationcited by 144

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 189; 2022: 687; 2023: 1223; 2024: 1658; 2025: 1716; 2026: 1420.

6,893 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  7. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
38 more
  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2021Scaling Vision Transformerscited by 993
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  6. 2023Vision Transformers Need Registerscited by 844
  7. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  8. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023Adversarial Diffusion Distillationcited by 741
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  13. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  15. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  16. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  17. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  18. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  19. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  22. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  23. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  24. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  25. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  27. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  28. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  29. 2023LERF: Language Embedded Radiance Fieldscited by 538
  30. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  31. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  32. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  33. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  34. 2021Intriguing Properties of Vision Transformerscited by 446
  35. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  36. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  37. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  38. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438

50 most cited of 6,893 All 6,893 (JSON)