mlsanityData to 7 Oct 2026

ResMLP: Feedforward networks for image classification with data-efficient training

arXiv 2105.03404 ↗First posted 7 May 2021cs.CVcited by 338

Not on the map. Why?

Abstract (arXiv)

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.

Key facts. Cited by 338 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

57 of 74 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2016Layer Normalizationcited by 5,579
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2019Searching for MobileNetV3cited by 1,984
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  13. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  14. 2017Random Erasing Data Augmentationcited by 1,235
  15. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  16. 2016Deep Networks with Stochastic Depthcited by 973
  17. 2020Designing Network Design Spacescited by 954
  18. 2018Image Transformercited by 770
  19. 2021Going deeper with Image Transformerscited by 759
  20. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  21. 2017Convolutional Sequence to Sequence Learningcited by 550
  22. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 497
  23. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  24. 2020Are we done with ImageNet?cited by 329
  25. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  26. 2020Exploring Self-attention for Image Recognitioncited by 313
  27. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  28. 2015Tensorizing Neural Networkscited by 246
  29. 2018Scaling Neural Machine Translationcited by 204
  30. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  31. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  32. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  33. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  34. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  35. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  36. 2020Towards Learning Convolutions from Scratchcited by 46
  37. 2021Fast and Accurate Model Scalingcited by 46
  38. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 91; 2023: 76; 2024: 62; 2025: 37; 2026: 24.

338 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  3. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  4. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  5. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  6. 2022DeiT III: Revenge of the ViTcited by 417
  7. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  8. 2021Pay Attention to MLPscited by 367
  9. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  10. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  11. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 301
  12. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
38 more
  1. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  2. 2021Global Filter Networks for Image Classificationcited by 257
  3. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  4. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  5. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  6. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  7. 2022Visual Attention Networkcited by 210
  8. 2022Patches Are All You Need?cited by 201
  9. 2022Focal Modulation Networkscited by 186
  10. 2022UNeXt: MLP-based Rapid Medical Image Segmentation Networkcited by 185
  11. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  12. 2022DaViT: Dual Attention Vision Transformerscited by 174
  13. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 158
  14. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  15. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  16. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  17. 2023TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecastingcited by 136
  18. 2022MetaFormer Baselines for Visioncited by 131
  19. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  20. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  21. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  22. 2021Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognitioncited by 103
  23. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  24. 2022Global Context Vision Transformerscited by 98
  25. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  26. 2022Three things everyone should know about Vision Transformerscited by 90
  27. 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
  28. 2022You Only Need 90K Parameters to Adapt Light: A Light Weight Transformer for Image Enhancement and Exposure Correctioncited by 81
  29. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  30. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  31. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  32. 2021PointMixer: MLP-Mixer for Point Cloud Understandingcited by 71
  33. 2021Hire-MLP: Vision MLP via Hierarchical Rearrangementcited by 66
  34. 2021PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifiercited by 63
  35. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  36. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 61
  37. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
  38. 2024SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detectioncited by 60

50 most cited of 338 All 338 (JSON)