mlsanityData to 7 Oct 2026

AutoFormer: Searching Transformers for Visual Recognition

arXiv 2107.00651 ↗First posted 1 Jul 2021cs.CVcited by 186

Not on the map. Why?

Abstract (arXiv)

Recently, pure transformer-based models have shown great potentials for vision tasks such as image classification and detection. However, the design of transformer networks is challenging. It has been observed that the depth, embedding dimension, and number of heads can largely affect the performance of vision transformers. Previous models configure these dimensions based upon manual crafting. In this work, we propose a new one-shot architecture search framework, namely AutoFormer, dedicated to vision transformer search. AutoFormer entangles the weights of different blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch. Besides, the searched models, which we refer to AutoFormers, surpass the recent state-of-the-arts such as ViT and DeiT. In particular, AutoFormer-tiny/small/base achieve 74.7%/81.7%/82.4% top-1 accuracy on ImageNet with 5.7M/22.9M/53.7M parameters, respectively. Lastly, we verify the transferability of AutoFormer by providing the performance on downstream benchmarks and distillation experiments. Code and models are available at https://github.com/microsoft/AutoML.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Builds on 63 of its 74 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

63 of 74 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2016Layer Normalizationcited by 5,579
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2019Searching for MobileNetV3cited by 1,984
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2021Transformers in Vision: A Surveycited by 1,009
  13. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2020Designing Network Design Spacescited by 954
  16. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  17. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  18. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  19. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  20. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  21. 2019Are Sixteen Heads Really Better than One?cited by 803
  22. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  23. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  24. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  25. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  26. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  27. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  28. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  29. 2014Deeply-Supervised Netscited by 364
  30. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  31. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  32. 2020Exploring Self-attention for Image Recognitioncited by 313
  33. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  34. 2018Slimmable Neural Networkscited by 301
  35. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  36. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  37. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  38. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 40; 2023: 41; 2024: 50; 2025: 32; 2026: 12.

186 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2022Transformers in Time Series: A Surveycited by 489
  3. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  4. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  5. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  6. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  8. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  9. 2022Neural Prompt Searchcited by 133
  10. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  11. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  12. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
38 more
  1. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  2. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 81
  3. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  4. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  5. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  6. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  7. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  8. 2023MixFormerV2: Efficient Fully Transformer Trackingcited by 56
  9. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  10. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  11. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  12. 2024A Survey on Transformer Compressioncited by 44
  13. 2023On Sparse Modern Hopfield Modelcited by 41
  14. 2021Searching the Search Space of Vision Transformercited by 39
  15. 2023A Survey on Efficient Training of Transformerscited by 37
  16. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  17. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  18. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  19. 2022Training-free Transformer Architecture Searchcited by 33
  20. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  21. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  22. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 32
  23. 2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
  24. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27
  25. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  26. 2021Sliced Recursive Transformercited by 26
  27. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  28. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  29. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  30. 2023Dynamic Grained Encoder for Vision Transformerscited by 24
  31. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 23
  32. 2023Large Pre-trained time series models for cross-domain Time series analysis taskscited by 22
  33. 2023CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 18
  34. 2023Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptationcited by 18
  35. 2024TimeRAG: BOOSTING LLM Time Series Forecasting via Retrieval-Augmented Generationcited by 18
  36. 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
  37. 2022Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessmentcited by 17
  38. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16

50 most cited of 186 All 186 (JSON)