mlsanityData to 7 Oct 2026

VanillaNet: the Power of Minimalism in Deep Learning

arXiv 2305.12972 ↗First posted 22 May 2023cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

50 of 64 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2017Mask R-CNNcited by 7,259
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  13. 2019Searching for MobileNetV3cited by 1,984
  14. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  15. 2017Random Erasing Data Augmentationcited by 1,235
  16. 2021Scaling Vision Transformerscited by 993
  17. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  18. 2021Going deeper with Image Transformerscited by 759
  19. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  20. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  21. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  22. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  23. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  24. 2020Are we done with ImageNet?cited by 329
  25. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  26. 2016Benefits of depth in neural networkscited by 248
  27. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  28. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  29. 2019ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blockscited by 178
  30. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  31. 2019Quantization Networkscited by 151
  32. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  33. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  34. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  35. 2021Diverse Branch Block: Building a Convolution as an Inception-like Unitcited by 87
  36. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  37. 2020Hit-Detector: Hierarchical Trinity Architecture Search for Object Detectioncited by 32
  38. 2021Universal Adder Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 10; 2025: 9.

23 papers.

Sort:

  1. 2024A Survey on Transformer Compressioncited by 44
  2. 2023SG-Former: Self-guided Transformer with Evolving Token Reallocationcited by 32
  3. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  4. 2025MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devicescited by 12
  5. 2024UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformercited by 9
  6. 2024LV-UNet: A Lightweight and Vanilla Model for Medical Image Segmentationcited by 6
  7. 2023MOFA: A Model Simplification Roadmap for Image Restoration on Mobile Devicescited by 5
  8. 2024StarLKNet: Star Mixup with Large Kernel Networks for Palm Vein Identificationcited by 5
  9. 2024ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effectcited by 4
  10. 2024Adapting LLaMA Decoder to Vision Transformercited by 3
  11. 2024SpiralMLP: A Lightweight Vision MLP Architecturecited by 1
  12. 2025HGO-YOLO: Advancing Anomaly Behavior Detection with Hierarchical Features and Lightweight Optimized Detectioncited by 1
11 more
  1. 2023Multi-view learning for automatic classification of multi-wavelength auroral imagescited by 0
  2. 2024RepAct: The Re-parameterizable Adaptive Activation Functioncited by 0
  3. 2024Activation function optimization method: Learnable series linear units (LSLUs)cited by 0
  4. 2024Designing Concise ConvNets with Columnar Stagescited by 0
  5. 2025UL-UNAS: Ultra-Lightweight U-Nets for Real-Time Speech Enhancement via Network Architecture Searchcited by 0
  6. 2025Partial Convolution Meets Visual Attentioncited by 0
  7. 2025A biologically inspired separable learning vision model for real-time traffic object perception in Darkcited by 0
  8. 2025TS-P2CL: Plug-and-Play Dual Contrastive Learning for Vision-Guided Medical Time Series Classificationcited by 0
  9. 2025A Hybrid Framework Bridging CNN and ViT based on Theory of Evidence for Diabetic Retinopathy Gradingcited by 0
  10. 2025Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systemscited by 0
  11. 2025Edge-Aware and Content-Adaptive Infrared Gas Leak Detection for Industrial Safety Monitoringcited by 0