mlsanityData to 14 Oct 2026

The Vanishing Decision Boundary Complexity and the Strong First Component

arXiv 2211.16209 ↗First posted 25 Nov 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 71 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

25 of 71 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  10. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  12. 2017Generalization in Deep Learningcited by 146
13 more
  1. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  2. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  3. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  4. 2021A Comprehensive Evaluation Framework for Deep Model Robustnesscited by 42
  5. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 40
  6. 2019Characterizing the Decision Boundary of Deep Neural Networkscited by 32
  7. 2020Hold me tight! Influence of discriminative features on deep network boundariescited by 31
  8. 2020On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspectivecited by 25
  9. 2020Understanding the Decision Boundary of Deep Neural Networks: An Empirical Studycited by 20
  10. 2020Analysis of Generalizability of Deep Neural Networks Based on the Complexity of Decision Boundarycited by 13
  11. 2022Understanding Deep Learning via Decision Boundarycited by 7
  12. 2020ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundariescited by 4
  13. 2022Class Interference of Deep Neural Networkscited by 1

Built on by

Not cited yet.