mlsanityData to 7 Oct 2026

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

arXiv 2101.11986 ↗First posted 28 Jan 2021cs.CVcited by 1,096

Not on the map. Why?

Cited by 29 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neighboring pixels, leading to low training sample efficiency; 2) the redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-ViT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study. Notably, T2T-ViT reduces the parameter count and MACs of vanilla ViT by half, while achieving more than 3.0% improvement when trained from scratch on ImageNet. It also outperforms ResNets and achieves comparable performance with MobileNets by directly training on ImageNet. For example, T2T-ViT with comparable size to ResNet50 (21.5M parameters) can achieve 83.3% top1 accuracy in image resolution 384×384 on ImageNet. (Code: https://github.com/yitu-opensource/T2T-ViT)

Key facts. Cited by 1,096 papers in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 28 Jan 2021.

Builds on

52 of 63 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2016Wide Residual Networkscited by 3,057
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  13. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  14. 2017Non-local Neural Networkscited by 2,290
  15. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  16. 2020Rethinking Attention with Performerscited by 1,584
  17. 2020Point Transformercited by 1,462
  18. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  19. 2020Pre-Trained Image Processing Transformercited by 891
  20. 2018Image Transformercited by 770
  21. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  22. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  23. 2019GhostNet: More Features from Cheap Operationscited by 410
  24. 2017Residual Attention Network for Image Classificationcited by 385
  25. 2019Attention Augmented Convolutional Networkscited by 363
  26. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  27. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  28. 2020Exploring Self-attention for Image Recognitioncited by 313
  29. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  30. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  31. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  32. 2019Local Relation Networks for Image Recognitioncited by 210
  33. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  34. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  35. 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 138
  36. 2018A2-Nets: Double Attention Networkscited by 136
  37. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  38. 2019Neural Epitome Search for Architecture-Agnostic Network Compressioncited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 223; 2022: 295; 2023: 252; 2024: 178; 2025: 112; 2026: 36.

1,096 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  5. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  7. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  8. 2021Scaling Vision Transformerscited by 993
  9. 2024VMamba: Visual State Space Modelcited by 955
  10. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  11. 2021Multiscale Vision Transformerscited by 896
  12. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
38 more
  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2021Going deeper with Image Transformerscited by 759
  5. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  8. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  9. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2021Transformer in Transformercited by 632
  12. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  13. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  14. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  15. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  16. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  17. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  18. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  19. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  20. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  21. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  22. 2021Intriguing Properties of Vision Transformerscited by 446
  23. 2022DeiT III: Revenge of the ViTcited by 417
  24. 2021Early Convolutions Help Transformers See Bettercited by 395
  25. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  26. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  27. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  28. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  29. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  30. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  31. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  32. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  33. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  34. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  35. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  36. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  37. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  38. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263

50 most cited of 1,096 All 1,096 (JSON)