# Auto-tuning of Deep Neural Networks by Conflicting Layer Removal

arXiv: https://arxiv.org/abs/2103.04331 · HTML: https://mlsanity.com/paper/2103.04331/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Data-dependent Initializations of Convolutional Neural Networks](https://mlsanity.com/paper/1511.06856/) (2015)
- [Norm-Preservation: Why Residual Networks Can Become Extremely Deep?](https://mlsanity.com/paper/1805.07477/) (2018)
- [Increasing the adversarial robustness and explainability of capsule networks with γ-capsules](https://mlsanity.com/paper/1812.09707/) (2018)

## Built on by (most cited first, 3 of 3)

- [Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization](https://mlsanity.com/paper/2208.01134/) (2022)
- [Training Deep Capsule Networks with Residual Connections](https://mlsanity.com/paper/2104.07393/) (2021)
- [Zero-Shot Neural Architecture Search with Weighted Response Correlation](https://mlsanity.com/paper/2507.08841/) (2025)

## Abstract (arXiv)

Designing neural network architectures is a challenging task and knowing which specific layers of a model must be adapted to improve the performance is almost a mystery. In this paper, we introduce a novel methodology to identify layers that decrease the test accuracy of trained models. Conflicting layers are detected as early as the beginning of training. In the worst-case scenario, we prove that such a layer could lead to a network that cannot be trained at all. A theoretical analysis is provided on what is the origin of those layers that result in a lower overall network performance, which is complemented by our extensive empirical evaluation. More precisely, we identified those layers that worsen the performance because they would produce what we name conflicting training bundles. We will show that around 60% of the layers of trained residual networks can be completely removed from the architecture with no significant increase in the test-error. We will further present a novel neural-architecture-search (NAS) algorithm that identifies conflicting layers at the beginning of the training. Architectures found by our auto-tuning algorithm achieve competitive accuracy values when compared against more complex state-of-the-art architectures, while drastically reducing memory consumption and inference time for different computer vision tasks. The source code is available on https://github.com/peerdavid/conflicting-bundles