mlsanityData to 7 Oct 2026

Non-deep Networks

arXiv 2110.07641 ↗First posted 14 Oct 2021cs.CV cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Depth is the hallmark of deep neural networks. But more depth means more sequential computation and higher latency. This begs the question -- is it possible to build high-performing "non-deep" neural networks? We show that it is. To do so, we use parallel subnetworks instead of stacking one layer after another. This helps effectively reduce depth while maintaining high performance. By utilizing parallel substructures, we show, for the first time, that a network with a depth of just 12 can achieve top-1 accuracy over 80% on ImageNet, 96% on CIFAR10, and 81% on CIFAR100. We also show that a network with a low-depth (12) backbone can achieve an AP of 48% on MS-COCO. We analyze the scaling rules for our design and show how to increase performance without changing the network's depth. Finally, we provide a proof of concept for how non-deep networks could be used to build low-latency recognition systems. Code is available at https://github.com/imankgoyal/NonDeepNetworks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

28 of 38 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
16 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  7. 2016Deep Networks with Stochastic Depthcited by 973
  8. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  9. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  10. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  11. 2020Scaled-YOLOv4: Scaling Cross Stage Partial Networkcited by 211
  12. 2015Attention to Scale: Scale-aware Semantic Image Segmentationcited by 169
  13. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  14. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  15. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  16. 2013Understanding Deep Architectures using a Recursive Convolutional Networkcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 2; 2024: 1; 2025: 3; 2026: 2.

14 papers.

Sort:

By area:

  1. 2022Three things everyone should know about Vision Transformerscited by 90
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  3. 2023Efficient Multi-Scale Attention Module with Cross-Spatial Learningcited by 40
  4. 2021Augmenting Convolutional networks with attention-based aggregationcited by 29
  5. 2022Dual Vision Transformercited by 29
  6. 2022Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammarscited by 9
  7. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  8. 2023DualAttNet: Synergistic Fusion of Image-level and Fine-Grained Disease Attention for Multi-Label Lesion Detection in Chest X-rayscited by 5
  9. 2021A Multi-channel Training Method Boost the Performancecited by 1
  10. 2024Designing Concise ConvNets with Columnar Stagescited by 0
  11. 2025Rethinking Vision Transformer Depth via Structural Reparameterizationcited by 0
  12. 2025Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systemscited by 0
2 more
  1. 2026Connections Between Pairs of Filters Improve the Accuracy of Convolutional Neural Networkscited by 0
  2. 2026Multi-view feature High-order Fusion for Space Weak Object Detection and Segmentationcited by 0