# Towards a universal mechanism for successful deep learning

arXiv: https://arxiv.org/abs/2309.07537 · HTML: https://mlsanity.com/paper/2309.07537/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent](https://mlsanity.com/paper/1607.01981/) (2016)
- [The mechanism underlying successful deep learning](https://mlsanity.com/paper/2305.18078/) (2023)
- [Enhancing the accuracies by performing pooling decisions adjacent to the output layer](https://mlsanity.com/paper/2303.05800/) (2023)
- [Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler](https://mlsanity.com/paper/2307.00198/) (2023)

## Built on by (most cited first, 8 of 8)

- [Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi](https://mlsanity.com/paper/2501.12900/) (2025)
- [Low-latency vision transformers via large-scale multi-head attention](https://mlsanity.com/paper/2506.23832/) (2025)
- [Tiny language models](https://mlsanity.com/paper/2507.14871/) (2025)
- [Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning](https://mlsanity.com/paper/2509.03407/) (2025)
- [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024)
- [Translation Entropy: A Statistical Framework for Evaluating Translation Systems](https://mlsanity.com/paper/2511.13180/) (2025)
- [Self-attention vector output similarities reveal how machines pay attention](https://mlsanity.com/paper/2512.21956/) (2025)
- [Single-Nodal Spontaneous Symmetry Breaking in NLP Models](https://mlsanity.com/paper/2601.20582/) (2026)

## Abstract (arXiv)

Recently, the underlying mechanism for successful deep learning (DL) was presented based on a quantitative method that measures the quality of a single filter in each layer of a DL model, particularly VGG-16 trained on CIFAR-10. This method exemplifies that each filter identifies small clusters of possible output labels, with additional noise selected as labels outside the clusters. This feature is progressively sharpened with each layer, resulting in an enhanced signal-to-noise ratio (SNR), which leads to an increase in the accuracy of the DL network. In this study, this mechanism is verified for VGG-16 and EfficientNet-B0 trained on the CIFAR-100 and ImageNet datasets, and the main results are as follows. First, the accuracy and SNR progressively increase with the layers. Second, for a given deep architecture, the maximal error rate increases approximately linearly with the number of output labels. Third, similar trends were obtained for dataset labels in the range [3, 1,000], thus supporting the universality of this mechanism. Understanding the performance of a single filter and its dominating features paves the way to highly dilute the deep architecture without affecting its overall accuracy, and this can be achieved by applying the filter's cluster connections (AFCC).