mlsanityData to 7 Oct 2026

Searching the Search Space of Vision Transformer

arXiv 2111.14725 ↗First posted 29 Nov 2021cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such as Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality to downstream vision and vision-language tasks. Code and models will be available at https://github.com/microsoft/Cream.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 55 of its 62 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

55 of 62 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2019Searching for MobileNetV3cited by 1,984
38 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  5. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  6. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  8. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2020Designing Network Design Spacescited by 954
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  13. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  14. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  15. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  16. 2017Progressive Neural Architecture Searchcited by 483
  17. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  18. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  19. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  20. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  21. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  22. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  23. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  24. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  25. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  26. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  27. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  28. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  29. 2019The Evolved Transformercited by 180
  30. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  31. 2017Genetic CNNcited by 124
  32. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  33. 2019NAS evaluation is frustratingly hardcited by 69
  34. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  35. 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 65
  36. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  37. 2020Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Searchcited by 38
  38. 2019XNAS: Neural Architecture Search with Expert Advicecited by 35

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 12; 2024: 7; 2025: 8; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  3. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  4. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  5. 2023CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 18
  6. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  7. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13
  8. 2023Automated Design of Metaheuristic Algorithms: A Surveycited by 12
  9. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  10. 2023ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devicescited by 10
  11. 2023MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memorycited by 9
  12. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
27 more
  1. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  2. 2023Explainable Knowledge Distillation for On-device Chest X-Ray Classificationcited by 7
  3. 2022Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Searchcited by 5
  4. 2022Lightweight Vision Transformer with Cross Feature Attentioncited by 5
  5. 2022FocusFormer: Focusing on What We Need via Architecture Samplercited by 5
  6. 2025Transferrable Surrogates in Expressive Neural Architecture Search Spacescited by 5
  7. 2022Similarity of Neural Architectures using Adversarial Attack Transferabilitycited by 4
  8. 2024DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisionscited by 4
  9. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AIcited by 3
  10. 2022Rethinking the Zigzag Flattening for Image Readingcited by 2
  11. 2023Rethink DARTS Search Space and Renovate a New Benchmarkcited by 2
  12. 2023Entropic Score metric: Decoupling Topology and Size in Training-free NAScited by 2
  13. 2024When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspectivecited by 2
  14. 2023SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inferencecited by 1
  15. 2024Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adaptercited by 1
  16. 2025CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inferencecited by 1
  17. 2023AutoTaskFormer: Searching Vision Transformers for Multi-task Learningcited by 0
  18. 2024SCAN-Edge: Finding MobileNet-speed Hybrid Networks for Diverse Edge Devices via Hardware-Aware Evolutionary Searchcited by 0
  19. 2025Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insightscited by 0
  20. 2025Input Resolution Downsizing as a Compression Technique for Vision Deep Learning Systemscited by 0
  21. 2025DASViT: Differentiable Architecture Search for Vision Transformercited by 0
  22. 2025Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Betweencited by 0
  23. 2025Hybrid Convolution and Vision Transformer NAS Search Space for TinyML Image Classificationcited by 0
  24. 2026Dual-Domain Representation Alignment: Bridging 2D and 3D Vision via Geometry-Aware Architecture Searchcited by 0
  25. 2026Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packagingcited by 0
  26. 2026Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Searchcited by 0
  27. 2026EVOTS: Evolutionary Transformer Search for Time Series Forecastingcited by 0