# VanillaNet: the Power of Minimalism in Deep Learning

arXiv: https://arxiv.org/abs/2305.12972 · HTML: https://mlsanity.com/paper/2305.12972/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 64 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)

## Built on by (most cited first, 23 of 23)

- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [SG-Former: Self-guided Transformer with Evolving Token Reallocation](https://mlsanity.com/paper/2308.12216/) (2023)
- [PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation](https://mlsanity.com/paper/2312.17276/) (2023)
- [MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices](https://mlsanity.com/paper/2507.01838/) (2025)
- [UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer](https://mlsanity.com/paper/2401.06426/) (2024)
- [LV-UNet: A Lightweight and Vanilla Model for Medical Image Segmentation](https://mlsanity.com/paper/2408.16886/) (2024)
- [MOFA: A Model Simplification Roadmap for Image Restoration on Mobile Devices](https://mlsanity.com/paper/2308.12494/) (2023)
- [StarLKNet: Star Mixup with Large Kernel Networks for Palm Vein Identification](https://mlsanity.com/paper/2405.12721/) (2024)
- [ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effect](https://mlsanity.com/paper/2401.12736/) (2024)
- [Adapting LLaMA Decoder to Vision Transformer](https://mlsanity.com/paper/2404.06773/) (2024)
- [SpiralMLP: A Lightweight Vision MLP Architecture](https://mlsanity.com/paper/2404.00648/) (2024)
- [HGO-YOLO: Advancing Anomaly Behavior Detection with Hierarchical Features and Lightweight Optimized Detection](https://mlsanity.com/paper/2503.07371/) (2025)
- [Multi-view learning for automatic classification of multi-wavelength auroral images](https://mlsanity.com/paper/2311.02947/) (2023)
- [RepAct: The Re-parameterizable Adaptive Activation Function](https://mlsanity.com/paper/2407.00131/) (2024)
- [Activation function optimization method: Learnable series linear units (LSLUs)](https://mlsanity.com/paper/2409.08283/) (2024)
- [Designing Concise ConvNets with Columnar Stages](https://mlsanity.com/paper/2410.04089/) (2024)
- [UL-UNAS: Ultra-Lightweight U-Nets for Real-Time Speech Enhancement via Network Architecture Search](https://mlsanity.com/paper/2503.00340/) (2025)
- [Partial Convolution Meets Visual Attention](https://mlsanity.com/paper/2503.03148/) (2025)
- [A biologically inspired separable learning vision model for real-time traffic object perception in Dark](https://mlsanity.com/paper/2509.05012/) (2025)
- [TS-P2CL: Plug-and-Play Dual Contrastive Learning for Vision-Guided Medical Time Series Classification](https://mlsanity.com/paper/2509.17802/) (2025)
- [A Hybrid Framework Bridging CNN and ViT based on Theory of Evidence for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2510.26315/) (2025)
- [Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systems](https://mlsanity.com/paper/2512.19731/) (2025)
- [Edge-Aware and Content-Adaptive Infrared Gas Leak Detection for Industrial Safety Monitoring](https://mlsanity.com/paper/2512.23234/) (2025)

## Abstract (arXiv)

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.