# Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

arXiv: https://arxiv.org/abs/1608.06037 · HTML: https://mlsanity.com/paper/1608.06037/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2016.

## Built on by (most cited first, 25 of 25)

- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence](https://mlsanity.com/paper/2105.01798/) (2021)
- [Mechanism of feature learning in convolutional neural networks](https://mlsanity.com/paper/2309.00570/) (2023)
- [The Manifold Hypothesis for Gradient-Based Explanations](https://mlsanity.com/paper/2206.07387/) (2022)
- [Similarity-based Label Inference Attack against Training and Inference of Split Learning](https://mlsanity.com/paper/2203.05222/) (2022)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [Model Doctor: A Simple Gradient Aggregation Strategy for Diagnosing and Treating CNN Classifiers](https://mlsanity.com/paper/2112.04934/) (2021)
- [ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions](https://mlsanity.com/paper/2109.04386/) (2021)
- [Generalization Bounds for Gradient Methods via Discrete and Continuous Prior](https://mlsanity.com/paper/2205.13799/) (2022)
- [Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets](https://mlsanity.com/paper/2304.02847/) (2023)
- [CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles](https://mlsanity.com/paper/2111.08243/) (2021)
- [DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators](https://mlsanity.com/paper/2412.06566/) (2024)
- [Synergy: Towards On-Body AI via Tiny AI Accelerator Collaboration on Wearables](https://mlsanity.com/paper/2401.08637/) (2023)
- [Ablation Studies for Novel Treatment Effect Estimation Models](https://mlsanity.com/paper/2410.15560/) (2024)
- [Genealogical Population-Based Training for Hyperparameter Optimization](https://mlsanity.com/paper/2109.14925/) (2021)
- [Pruning Distorted Images in MNIST Handwritten Digits](https://mlsanity.com/paper/2307.14343/) (2023)
- [An Approach for Efficient Neural Architecture Search Space Definition](https://mlsanity.com/paper/2310.17669/) (2023)
- [FPGA-QHAR: Throughput-Optimized for Quantized Human Action Recognition on The Edge](https://mlsanity.com/paper/2311.03390/) (2023)
- [Improved Background Estimation for Gas Plume Identification in Hyperspectral Images](https://mlsanity.com/paper/2411.15378/) (2024)
- [Attackers Can Do Better: Over- and Understated Factors of Model Stealing Attacks](https://mlsanity.com/paper/2503.06188/) (2025)
- [LightPneumoNet: Lightweight Pneumonia Classifier](https://mlsanity.com/paper/2510.11232/) (2025)
- [ArtNet: Hierarchical Clustering-Based Artificial Netlist Generator for ML and DTCO Application](https://mlsanity.com/paper/2510.13582/) (2025)
- [Growing Networks with Autonomous Pruning](https://mlsanity.com/paper/2603.19759/) (2026)
- [Restoring Neural Network Plasticity for Faster Transfer Learning](https://mlsanity.com/paper/2603.20860/) (2026)
- [Deep-testing: the case of dependence detection](https://mlsanity.com/paper/2604.26558/) (2026)

## Abstract (arXiv)

Major winning Convolutional Neural Networks (CNNs), such as AlexNet, VGGNet, ResNet, GoogleNet, include tens to hundreds of millions of parameters, which impose considerable computation and memory overhead. This limits their practical use for training, optimization and memory efficiency. On the contrary, light-weight architectures, being proposed to address this issue, mainly suffer from low accuracy. These inefficiencies mostly stem from following an ad hoc procedure. We propose a simple architecture, called SimpleNet, based on a set of designing principles, with which we empirically show, a well-crafted yet simple and reasonably deep architecture can perform on par with deeper and more complex architectures. SimpleNet provides a good tradeoff between the computation/memory efficiency and the accuracy. Our simple 13-layer architecture outperforms most of the deeper and complex architectures to date such as VGGNet, ResNet, and GoogleNet on several well-known benchmarks while having 2 to 25 times fewer number of parameters and operations. This makes it very handy for embedded systems or systems with computational and memory limitations. We achieved state-of-the-art result on CIFAR10 outperforming several heavier architectures, near state of the art on MNIST and competitive results on CIFAR100 and SVHN. We also outperformed the much larger and deeper architectures such as VGGNet and popular variants of ResNets among others on the ImageNet dataset. Models are made available at: https://github.com/Coderx7/SimpleNet