# The Vanishing Decision Boundary Complexity and the Strong First Component

arXiv: https://arxiv.org/abs/2211.16209 · HTML: https://mlsanity.com/paper/2211.16209/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 71 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [Asymmetric Valleys: Beyond Sharp and Flat Local Minima](https://mlsanity.com/paper/1902.00744/) (2019)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [Knowledge Distillation with Adversarial Samples Supporting Decision Boundary](https://mlsanity.com/paper/1805.05532/) (2018)
- [A Comprehensive Evaluation Framework for Deep Model Robustness](https://mlsanity.com/paper/2101.09617/) (2021)
- [Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples](https://mlsanity.com/paper/2111.02625/) (2021)
- [Characterizing the Decision Boundary of Deep Neural Networks](https://mlsanity.com/paper/1912.11460/) (2019)
- [Hold me tight! Influence of discriminative features on deep network boundaries](https://mlsanity.com/paper/2002.06349/) (2020)
- [On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective](https://mlsanity.com/paper/2002.08838/) (2020)
- [Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study](https://mlsanity.com/paper/2002.01810/) (2020)
- [Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundary](https://mlsanity.com/paper/2009.07974/) (2020)
- [Understanding Deep Learning via Decision Boundary](https://mlsanity.com/paper/2206.01515/) (2022)
- [ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundaries](https://mlsanity.com/paper/2012.10282/) (2020)
- [Class Interference of Deep Neural Networks](https://mlsanity.com/paper/2211.01370/) (2022)

## Abstract (arXiv)

We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.