# Pay Attention to MLPs

arXiv: https://arxiv.org/abs/2105.08050 · HTML: https://mlsanity.com/paper/2105.08050/

## Key facts

- Cited by 368 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 51 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)

## Built on by (most cited first, 30 of 368)

- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [MAXIM: Multi-Axis MLP for Image Processing](https://mlsanity.com/paper/2201.02973/) (2022)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [Transformer Quality in Linear Time](https://mlsanity.com/paper/2202.10447/) (2022)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [MixVPR: Feature Mixing for Visual Place Recognition](https://mlsanity.com/paper/2303.02190/) (2023)
- [Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures](https://mlsanity.com/paper/2207.01186/) (2022)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity](https://mlsanity.com/paper/2207.03620/) (2022)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)
- [Dual Aggregation Transformer for Image Super-Resolution](https://mlsanity.com/paper/2308.03364/) (2023)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [An Attention Free Transformer](https://mlsanity.com/paper/2105.14103/) (2021)

## Abstract (arXiv)

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.