# Efficient shallow learning as an alternative to deep learning

arXiv: https://arxiv.org/abs/2211.11106 · HTML: https://mlsanity.com/paper/2211.11106/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent](https://mlsanity.com/paper/1607.01981/) (2016)
- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet](https://mlsanity.com/paper/1802.06205/) (2018)
- [Brain experiments imply adaptation mechanisms which outperform common AI learning algorithms](https://mlsanity.com/paper/2005.04106/) (2020)
- [Brain inspired neuronal silencing mechanism to enable reliable sequence identification](https://mlsanity.com/paper/2203.13028/) (2022)

## Built on by (most cited first, 6 of 6)

- [Low-latency vision transformers via large-scale multi-head attention](https://mlsanity.com/paper/2506.23832/) (2025)
- [Enhancing the accuracies by performing pooling decisions adjacent to the output layer](https://mlsanity.com/paper/2303.05800/) (2023)
- [Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning](https://mlsanity.com/paper/2509.03407/) (2025)
- [Statistical Mechanics of Learning via Reverberation in Bidirectional Associative Memories](https://mlsanity.com/paper/2307.08365/) (2023)
- [Make Deep Networks Shallow Again](https://mlsanity.com/paper/2309.08414/) (2023)
- [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024)

## Abstract (arXiv)

The realization of complex classification tasks requires training of deep learning (DL) architectures consisting of tens or even hundreds of convolutional and fully connected hidden layers, which is far from the reality of the human brain. According to the DL rationale, the first convolutional layer reveals localized patterns in the input and large-scale patterns in the following layers, until it reliably characterizes a class of inputs. Here, we demonstrate that with a fixed ratio between the depths of the first and second convolutional layers, the error rates of the generalized shallow LeNet architecture, consisting of only five layers, decay as a power law with the number of filters in the first convolutional layer. The extrapolation of this power law indicates that the generalized LeNet can achieve small error rates that were previously obtained for the CIFAR-10 database using DL architectures. A power law with a similar exponent also characterizes the generalized VGG-16 architecture. However, this results in a significantly increased number of operations required to achieve a given error rate with respect to LeNet. This power law phenomenon governs various generalized LeNet and VGG-16 architectures, hinting at its universal behavior and suggesting a quantitative hierarchical time-space complexity among machine learning architectures. Additionally, the conservation law along the convolutional layers, which is the square-root of their size times their depth, is found to asymptotically minimize error rates. The efficient shallow learning that is demonstrated in this study calls for further quantitative examination using various databases and architectures and its accelerated implementation using future dedicated hardware developments.