mlsanityData to 7 Oct 2026

A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP

arXiv 2108.13002 ↗First posted 30 Aug 2021cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision. Recently, Transformer and multi-layer perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer, started to lead new trends as they showed promising results in the ImageNet classification task. In this paper, we conduct empirical studies on these DNN structures and try to understand their respective pros and cons. To ensure a fair comparison, we first develop a unified framework called SPACH which adopts separate modules for spatial and channel processing. Our experiments under the SPACH framework reveal that all structures can achieve competitive performance at a moderate scale. However, they demonstrate distinctive behaviors when the network size scales up. Based on our findings, we propose two hybrid models using convolution and Transformer modules. The resulting Hybrid-MS-S+ model achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is already on par with the SOTA models with sophisticated designs. The code and models are publicly available at https://github.com/microsoft/SPACH.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 30 Aug 2021.

Builds on

37 of 42 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2017Mask R-CNNcited by 7,259
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
25 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2017Random Erasing Data Augmentationcited by 1,235
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2020Designing Network Design Spacescited by 954
  13. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  14. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  15. 2021Going deeper with Image Transformerscited by 759
  16. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  17. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  18. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  19. 2021Pay Attention to MLPscited by 367
  20. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  21. 2021Global Filter Networks for Image Classificationcited by 257
  22. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  23. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  24. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  25. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 6; 2024: 7; 2025: 6; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  2. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  3. 2022MetaFormer Baselines for Visioncited by 131
  4. 2022Inception Transformercited by 99
  5. 2023HAT: Hybrid Attention Transformer for Image Restorationcited by 64
  6. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  7. 2022Peripheral Vision Transformercited by 27
  8. 2023MLPST: MLP is All You Need for Spatio-Temporal Predictioncited by 22
  9. 2024Activating Wider Areas in Image Super-Resolutioncited by 10
  10. 2022LighTN: Light-weight Transformer Network for Performance-overhead Tradeoff in Point Cloud Downsamplingcited by 8
  11. 2022MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognitioncited by 8
  12. 2022CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidancecited by 7
20 more
  1. 2023Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inferencecited by 7
  2. 2021Hformer: Hybrid CNN-Transformer for Fringe Order Prediction in Phase Unwrapping of Fringe Projectioncited by 3
  3. 2024Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applicationscited by 3
  4. 2024Exploring Real&Synthetic Dataset and Linear Attention in Image Restorationcited by 3
  5. 2025Joint Resource Management for Energy-efficient UAV-assisted SWIPT-MEC: A Deep Reinforcement Learning Approachcited by 3
  6. 2022Accelerating the Training of Video Super-Resolution Modelscited by 2
  7. 2023SkyGPT: Probabilistic Short-term Solar Forecasting Using Synthetic Sky Videos from Physics-constrained VideoGPTcited by 2
  8. 2023MLP-AIR: An Efficient MLP-Based Method for Actor Interaction Relation Learning in Group Activity Recognitioncited by 1
  9. 2024FFNet: MetaMixer-based Efficient Convolutional Mixer Designcited by 1
  10. 2022Image Semantic Relation Generationcited by 0
  11. 2023SCSC: Spatial Cross-scale Convolution Module to Strengthen both CNNs and Transformerscited by 0
  12. 2024Knowledge Translation: A New Pathway for Model Compressioncited by 0
  13. 2024Ophthalmic Biomarker Detection with Parallel Prediction of Transformer and Convolutional Architecturecited by 0
  14. 2024DBF-Net: A Dual-Branch Network with Feature Fusion for Ultrasound Image Segmentationcited by 0
  15. 2025A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognitioncited by 0
  16. 2025Spectral-Adaptive Modulation Networks for Visual Perceptioncited by 0
  17. 2025Exploring Synergistic Ensemble Learning: Uniting CNNs, MLP-Mixers, and Vision Transformers to Enhance Image Classificationcited by 0
  18. 2025DetailSemNet: Elevating Signature Verification through Detail-Semantic Integrationcited by 0
  19. 2025Selective Masking based Self-Supervised Learning for Image Semantic Segmentationcited by 0
  20. 2026BMCR: Adaptive Backbone Module Composition via Reinforcement Learning for Remote Sensing Object Detectioncited by 0