mlsanityData to 7 Oct 2026

Efficient Self-supervised Vision Transformers for Representation Learning

arXiv 2106.09785 ↗First posted 17 Jun 2021cs.CV cs.AI cs.LGcited by 149

Not on the map. Why?

Abstract (arXiv)

This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models are publicly available: https://github.com/microsoft/esvit

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 76 of its 85 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

76 of 85 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2017Decoupled Weight Decay Regularizationcited by 1,722
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  16. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  17. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  18. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  19. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  20. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  21. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  22. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  23. 2016Colorful Image Colorizationcited by 1,113
  24. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  25. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  26. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  27. 2020Pre-Trained Image Processing Transformercited by 891
  28. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  29. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  30. 2018Image Transformercited by 770
  31. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  32. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  33. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  34. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  35. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  36. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  37. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  38. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 58; 2023: 44; 2024: 25; 2025: 9; 2026: 4.

149 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  3. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  4. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  5. 2023A Cookbook of Self-Supervised Learningcited by 248
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  8. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  9. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  10. 2022FlexiViT: One Model for All Patch Sizescited by 151
  11. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  12. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
38 more
  1. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  2. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  3. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  4. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  5. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  6. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  7. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  8. 2022Self-Supervised Visual Representation Learning with Semantic Groupingcited by 70
  9. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  10. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  11. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60
  12. 2023Diffusion Models Beat GANs on Image Classificationcited by 57
  13. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  14. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 47
  15. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 46
  16. 2023A Closer Look at Few-shot Classification Againcited by 46
  17. 2022Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentationcited by 41
  18. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
  19. 2022Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performancecited by 40
  20. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  21. 2021TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentationcited by 38
  22. 2022Rethinking Generalization in Few-Shot Classificationcited by 36
  23. 2021A Survey of Self-Supervised and Few-Shot Object Detectioncited by 35
  24. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 34
  25. 2023Visual Tuningcited by 34
  26. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
  27. 2023Model Spider: Learning to Rank Pre-Trained Models Efficientlycited by 32
  28. 2021Active Learning at the ImageNet Scalecited by 28
  29. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  30. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  31. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  32. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  33. 2021Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancerscited by 23
  34. 2022HiViT: Hierarchical Vision Transformer Meets Masked Image Modelingcited by 23
  35. 2022Revisiting a kNN-based Image Classification System with High-capacity Storagecited by 22
  36. 2022Evaluating Self-Supervised Learning for Molecular Graph Embeddingscited by 22
  37. 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
  38. 2022Exploring Long-Sequence Masked Autoencoderscited by 22

50 most cited of 149 All 149 (JSON)