mlsanityData to 7 Oct 2026

Semi-Supervised Vision Transformers

arXiv 2111.11067 ↗First posted 22 Nov 2021cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

We study the training of Vision Transformers for semi-supervised image classification. Transformers have recently demonstrated impressive performance on a multitude of supervised learning tasks. Surprisingly, we show Vision Transformers perform significantly worse than Convolutional Neural Networks when only a small set of labeled data is available. Inspired by this observation, we introduce a joint semi-supervised learning framework, Semiformer, which contains a transformer stream, a convolutional stream and a carefully designed fusion module for knowledge sharing between these streams. The convolutional stream is trained on limited labeled data and further used to generate pseudo labels to supervise the training of the transformer stream on unlabeled data. Extensive experiments on ImageNet demonstrate that Semiformer achieves 75.5% top-1 accuracy, outperforming the state-of-the-art by a clear margin. In addition, we show, among other things, Semiformer is a general framework that is compatible with most modern transformer and convolutional neural architectures. Code is available at https://github.com/wengzejia1/Semiformer.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

31 of 42 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
19 more
  1. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  2. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  6. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  7. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  8. 2021Early Convolutions Help Transformers See Bettercited by 395
  9. 2020Meta Pseudo Labelscited by 372
  10. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  11. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  12. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  13. 2015Semi-Supervised Learning with Ladder Networkscited by 246
  14. 2014Learning with Pseudo-Ensemblescited by 241
  15. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  16. 2021Conformer: Local Features Coupling Global Representations for Visual Recognitioncited by 164
  17. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  18. 2020Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptationcited by 41
  19. 2020Milking CowMask for Semi-Supervised Image Classificationcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 3; 2025: 4; 2026: 2.

24 papers.

Sort:

By area:

  1. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  2. 2023FoundPose: Unseen Object Pose Estimation with Foundation Featurescited by 109
  3. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 67
  4. 2022Prototypical Residual Networks for Anomaly Detection and Localizationcited by 56
  5. 2022Semi-supervised Vision Transformers at Scalecited by 49
  6. 2022ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localizationcited by 39
  7. 2025TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysiscited by 35
  8. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  9. 2023Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentationcited by 24
  10. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  11. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 21
  12. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
12 more
  1. 2022Semi-Supervised Single-View 3D Reconstruction via Prototype Shape Priorscited by 14
  2. 2023SimMatchV2: Semi-Supervised Learning with Graph Consistencycited by 12
  3. 2023Semi-Supervised Semantic Segmentation via Marginal Contextual Informationcited by 8
  4. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  5. 2025TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learningcited by 4
  6. 2024Cross Branch Feature Fusion Decoder for Consistency Regularization-based Semi-Supervised Change Detectioncited by 2
  7. 2026Triple Expert Learning from Noisy Labels for Semi-Supervised Vision Foundation Model Adaptationcited by 2
  8. 2023Semi-MAE: Masked Autoencoders for Semi-supervised Vision Transformerscited by 1
  9. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-trainingcited by 1
  10. 2023Transformadores: Fundamentos teoricos y Aplicacionescited by 0
  11. 2025SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learningcited by 0
  12. 2026Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Datacited by 0