mlsanityData to 7 Oct 2026

Pay Attention to MLPs

arXiv 2105.08050 ↗First posted 17 May 2021cs.LG cs.CL cs.CVcited by 367

Not on the map. Why?

Abstract (arXiv)

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

Key facts. Cited by 367 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
32 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  13. 2020GLU Variants Improve Transformercited by 1,724
  14. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  16. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  17. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  18. 2016Deep Networks with Stochastic Depthcited by 973
  19. 2020Designing Network Design Spacescited by 954
  20. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  21. 2016Language Modeling with Gated Convolutional Networkscited by 688
  22. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  23. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  24. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  25. 2015Highway Networkscited by 257
  26. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  27. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  28. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  29. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  30. 2020Toward Transformer-Based Object Detectioncited by 84
  31. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  32. 2021RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognitioncited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 99; 2023: 65; 2024: 77; 2025: 46; 2026: 33.

367 papers.

Sort:

By area:

  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  4. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  5. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 301
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  7. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  8. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  9. 2021Global Filter Networks for Image Classificationcited by 257
  10. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  11. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  12. 2022Visual Attention Networkcited by 210
38 more
  1. 2022Patches Are All You Need?cited by 201
  2. 2022Transformer Quality in Linear Timecited by 189
  3. 2021Are Transformers More Robust Than CNNs?cited by 187
  4. 2022Focal Modulation Networkscited by 186
  5. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  6. 2022DaViT: Dual Attention Vision Transformerscited by 174
  7. 2023MixVPR: Feature Mixing for Visual Place Recognitioncited by 160
  8. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 158
  9. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  10. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  11. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  12. 2023TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecastingcited by 136
  13. 2022MetaFormer Baselines for Visioncited by 131
  14. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  15. 2023Dual Aggregation Transformer for Image Super-Resolutioncited by 124
  16. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  17. 2024Transformers are Multi-State RNNscited by 120
  18. 2021An Attention Free Transformercited by 111
  19. 2021Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognitioncited by 103
  20. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  21. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  22. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  23. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  24. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  25. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  26. 2022Three things everyone should know about Vision Transformerscited by 90
  27. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  28. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  29. 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
  30. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  31. 2021PointMixer: MLP-Mixer for Point Cloud Understandingcited by 71
  32. 2022Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentationcited by 71
  33. 2021Hire-MLP: Vision MLP via Hierarchical Rearrangementcited by 66
  34. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  35. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  36. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  37. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
  38. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61

50 most cited of 367 All 367 (JSON)