mlsanityData to 14 Oct 2026

MetaFormer Is Actually What You Need for Vision

arXiv 2111.11418 ↗First posted 22 Nov 2021cs.CV cs.AI cs.LGcited by 524

Not on the map. Why?

Cited by 25 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in Transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in Transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned Vision Transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 50%/62% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of "MetaFormer", a general architecture abstracted from Transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent Transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design. Code is available at https://github.com/sail-sg/poolformer.

Key facts. Cited by 524 papers in the map (data through 14 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

54 of 70 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2017Mask R-CNNcited by 7,277
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
38 more
  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2016Layer Normalizationcited by 5,669
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  13. 2018Group Normalizationcited by 1,524
  14. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  15. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  16. 2017Random Erasing Data Augmentationcited by 1,239
  17. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  18. 2016Deep Networks with Stochastic Depthcited by 979
  19. 2020Designing Network Design Spacescited by 960
  20. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  21. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  22. 2021Going deeper with Image Transformerscited by 764
  23. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  24. 2019Panoptic Feature Pyramid Networkscited by 645
  25. 2021Transformer in Transformercited by 633
  26. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  27. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  28. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  29. 2021Pay Attention to MLPscited by 368
  30. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  31. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  32. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 339
  33. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  34. 2021Global Filter Networks for Image Classificationcited by 260
  35. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  36. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  37. 2018A2-Nets: Double Attention Networkscited by 136
  38. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 85; 2023: 134; 2024: 148; 2025: 98; 2026: 55.

524 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  3. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  4. 2022How Do Vision Transformers Work?cited by 307
  5. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  6. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  7. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 224
  8. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  9. 2022Visual Attention Networkcited by 210
  10. 2022Patches Are All You Need?cited by 201
  11. 2022Focal Modulation Networkscited by 186
  12. 2022Vision GNN: An Image is Worth Graph of Nodescited by 186
38 more
  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  2. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 174
  3. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  4. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  5. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 160
  6. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  7. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 143
  8. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 143
  9. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  10. 2024InstructIR: High-Quality Image Restoration Following Human Instructionscited by 140
  11. 2022MetaFormer Baselines for Visioncited by 132
  12. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 130
  13. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  14. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  15. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  16. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  17. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  18. 2024Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speedcited by 120
  19. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  20. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  21. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  22. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  23. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  24. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  25. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 103
  26. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  27. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  28. 2022Inception Transformercited by 99
  29. 2022Global Context Vision Transformerscited by 98
  30. 2023Large Selective Kernel Network for Remote Sensing Object Detectioncited by 98
  31. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  32. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  33. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  34. 2023InceptionNeXt: When Inception Meets ConvNeXtcited by 86
  35. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  36. 2023Energy Transformercited by 71
  37. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  38. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 66

50 most cited of 524 All 524 (JSON)