mlsanityData to 14 Oct 2026

All Tokens Matter: Token Labeling for Training Better Vision Transformers

arXiv 2104.10858 ↗First posted 22 Apr 2021cs.CVcited by 144

Not on the map. Why?

Abstract (arXiv)

In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pre-trained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and all the training details will be made publicly available at https://github.com/zihangJiang/TokenLabeling.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

54 of 63 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2017Mask R-CNNcited by 7,277
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2016Pyramid Scene Parsing Networkcited by 2,479
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  10. 2020Point Transformercited by 1,470
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  13. 2017Random Erasing Data Augmentationcited by 1,239
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  15. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  16. 2016Deep Networks with Stochastic Depthcited by 979
  17. 2020Designing Network Design Spacescited by 960
  18. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  19. 2020Pre-Trained Image Processing Transformercited by 893
  20. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  21. 2018Image Transformercited by 772
  22. 2021Going deeper with Image Transformerscited by 764
  23. 2021Transformer in Transformercited by 633
  24. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  25. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  26. 2020ResNeSt: Split-Attention Networkscited by 445
  27. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  28. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  29. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  30. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  31. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  32. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 347
  33. 2020Are we done with ImageNet?cited by 333
  34. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  35. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  36. 2019Fixing the train-test resolution discrepancycited by 209
  37. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 163
  38. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 47; 2023: 40; 2024: 27; 2025: 9; 2026: 4.

144 papers.

Sort:

By area:

  1. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  2. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  3. 2022Neighborhood Attention Transformercited by 239
  4. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  5. 2022Focal Modulation Networkscited by 186
  6. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  8. 2022DaViT: Dual Attention Vision Transformerscited by 175
  9. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  10. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  11. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 151
  12. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
38 more
  1. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  2. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  3. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  4. 2021A Survey of Visual Transformerscited by 117
  5. 2022Inception Transformercited by 99
  6. 2022OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mappingcited by 82
  7. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  8. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  9. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  10. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  11. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  12. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 76
  13. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  14. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 65
  15. 2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
  16. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  17. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  18. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  19. 2022CamoFormer: Masked Separable Attention for Camouflaged Object Detectioncited by 58
  20. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  21. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  22. 2023SpectFormer: Frequency and Attention is what you need in a Vision Transformercited by 48
  23. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  24. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  25. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  26. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  27. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39
  28. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 38
  29. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
  30. 2021Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attentioncited by 36
  31. 2023Rethinking Mobile Block for Efficient Attention-based Modelscited by 36
  32. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  33. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  34. 2022Vision Transformer with Attentive Pooling for Robust Facial Expression Recognitioncited by 35
  35. 2023DilateFormer: Multi-Scale Dilated Transformer for Visual Recognitioncited by 34
  36. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  37. 2021A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLPcited by 32
  38. 2022Dual Vision Transformercited by 29

50 most cited of 144 All 144 (JSON)