# A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP

arXiv: https://arxiv.org/abs/2108.13002 · HTML: https://mlsanity.com/paper/2108.13002/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 42 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)

## Built on by (most cited first, 30 of 32)

- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Inception Transformer](https://mlsanity.com/paper/2205.12956/) (2022)
- [HAT: Hybrid Attention Transformer for Image Restoration](https://mlsanity.com/paper/2309.05239/) (2023)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [Peripheral Vision Transformer](https://mlsanity.com/paper/2206.06801/) (2022)
- [MLPST: MLP is All You Need for Spatio-Temporal Prediction](https://mlsanity.com/paper/2309.13363/) (2023)
- [Activating Wider Areas in Image Super-Resolution](https://mlsanity.com/paper/2403.08330/) (2024)
- [LighTN: Light-weight Transformer Network for Performance-overhead Tradeoff in Point Cloud Downsampling](https://mlsanity.com/paper/2202.06263/) (2022)
- [MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognition](https://mlsanity.com/paper/2209.00407/) (2022)
- [CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance](https://mlsanity.com/paper/2203.09887/) (2022)
- [Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference](https://mlsanity.com/paper/2306.12465/) (2023)
- [Hformer: Hybrid CNN-Transformer for Fringe Order Prediction in Phase Unwrapping of Fringe Projection](https://mlsanity.com/paper/2112.06759/) (2021)
- [Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applications](https://mlsanity.com/paper/2403.17787/) (2024)
- [Exploring Real&Synthetic Dataset and Linear Attention in Image Restoration](https://mlsanity.com/paper/2412.03814/) (2024)
- [Joint Resource Management for Energy-efficient UAV-assisted SWIPT-MEC: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2505.03230/) (2025)
- [Accelerating the Training of Video Super-Resolution Models](https://mlsanity.com/paper/2205.05069/) (2022)
- [SkyGPT: Probabilistic Short-term Solar Forecasting Using Synthetic Sky Videos from Physics-constrained VideoGPT](https://mlsanity.com/paper/2306.11682/) (2023)
- [MLP-AIR: An Efficient MLP-Based Method for Actor Interaction Relation Learning in Group Activity Recognition](https://mlsanity.com/paper/2304.08803/) (2023)
- [FFNet: MetaMixer-based Efficient Convolutional Mixer Design](https://mlsanity.com/paper/2406.02021/) (2024)
- [Image Semantic Relation Generation](https://mlsanity.com/paper/2210.11253/) (2022)
- [SCSC: Spatial Cross-scale Convolution Module to Strengthen both CNNs and Transformers](https://mlsanity.com/paper/2308.07110/) (2023)
- [Knowledge Translation: A New Pathway for Model Compression](https://mlsanity.com/paper/2401.05772/) (2024)
- [Ophthalmic Biomarker Detection with Parallel Prediction of Transformer and Convolutional Architecture](https://mlsanity.com/paper/2409.17788/) (2024)
- [DBF-Net: A Dual-Branch Network with Feature Fusion for Ultrasound Image Segmentation](https://mlsanity.com/paper/2411.11116/) (2024)
- [A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition](https://mlsanity.com/paper/2502.16762/) (2025)
- [Spectral-Adaptive Modulation Networks for Visual Perception](https://mlsanity.com/paper/2503.23947/) (2025)
- [Exploring Synergistic Ensemble Learning: Uniting CNNs, MLP-Mixers, and Vision Transformers to Enhance Image Classification](https://mlsanity.com/paper/2504.09076/) (2025)
- [DetailSemNet: Elevating Signature Verification through Detail-Semantic Integration](https://mlsanity.com/paper/2511.16364/) (2025)

## Abstract (arXiv)

Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision. Recently, Transformer and multi-layer perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer, started to lead new trends as they showed promising results in the ImageNet classification task. In this paper, we conduct empirical studies on these DNN structures and try to understand their respective pros and cons. To ensure a fair comparison, we first develop a unified framework called SPACH which adopts separate modules for spatial and channel processing. Our experiments under the SPACH framework reveal that all structures can achieve competitive performance at a moderate scale. However, they demonstrate distinctive behaviors when the network size scales up. Based on our findings, we propose two hybrid models using convolution and Transformer modules. The resulting Hybrid-MS-S+ model achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is already on par with the SOTA models with sophisticated designs. The code and models are publicly available at https://github.com/microsoft/SPACH.