mlsanityData to 7 Oct 2026

Do Vision Transformers See Like Convolutional Neural Networks?

arXiv 2108.08810 ↗First posted 19 Aug 2021cs.CV cs.AI cs.LGcited by 671

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Key facts. Cited by 671 papers in the map (data through 7 Oct 2026). Builds on 39 of its 61 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

39 of 61 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
27 more
  1. 2018Image Transformercited by 770
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  3. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  4. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  5. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  7. 2021Intriguing Properties of Vision Transformerscited by 446
  8. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  9. 2019Attention Augmented Convolutional Networkscited by 363
  10. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  11. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  12. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  13. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  14. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  15. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  16. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  17. 2021Vision Transformers are Robust Learnerscited by 208
  18. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  19. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  20. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  21. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  22. 2020Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Futurecited by 83
  23. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  24. 2019Universality and individuality in neural dynamics across large populations of recurrent networkscited by 61
  25. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  26. 2021Supervised Transfer Learning at Scale for Medical Imagingcited by 38
  27. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 144; 2023: 137; 2024: 113; 2025: 118; 2026: 129.

671 papers.

Sort:

By area:

  1. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  2. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  3. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  4. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  5. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  6. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  7. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  8. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  11. 2022How Do Vision Transformers Work?cited by 305
  12. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
38 more
  1. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  2. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  3. 2022Multi-Game Decision Transformerscited by 212
  4. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  5. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  6. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  8. 2023Language-Driven Representation Learning for Roboticscited by 163
  9. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  10. 2022Understanding The Robustness in Vision Transformerscited by 144
  11. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  12. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  13. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  14. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  15. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  16. 2023CellViT: Vision Transformers for Precise Cell Segmentation and Classificationcited by 108
  17. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  18. 2022Inception Transformercited by 99
  19. 2022Global Context Vision Transformerscited by 98
  20. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  21. 2021Multi-Scale High-Resolution Vision Transformer for Semantic Segmentationcited by 92
  22. 2022Vision Transformers provably learn spatial structurecited by 92
  23. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  24. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  25. 2021Intriguing Findings of Frequency Selection for Image Deblurringcited by 80
  26. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  27. 2022Unified Visual Transformer Compressioncited by 79
  28. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  29. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  30. 2022Human alignment of neural network representationscited by 74
  31. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  32. 2022LargeKernel3D: Scaling up Kernels in 3D Sparse CNNscited by 71
  33. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  34. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  35. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  36. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  37. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  38. 2021On Efficient Transformer-Based Image Pre-training for Low-Level Visioncited by 64

50 most cited of 671 All 671 (JSON)