# On the Decision Boundary of Deep Neural Networks

arXiv: https://arxiv.org/abs/1808.05385 · HTML: https://mlsanity.com/paper/1808.05385/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2018.

## Built on by (most cited first, 11 of 11)

- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary](https://mlsanity.com/paper/2505.18325/) (2025)
- [BinarizedAttack: Structural Poisoning Attacks to Graph-based Anomaly Detection](https://mlsanity.com/paper/2106.09989/) (2021)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [Relevant Irrelevance: Generating Alterfactual Explanations for Image Classifiers](https://mlsanity.com/paper/2405.05295/) (2024)
- [LADDER: Latent Boundary-guided Adversarial Training](https://mlsanity.com/paper/2206.03717/) (2022)
- [AutoML for Multi-Class Anomaly Compensation of Sensor Drift](https://mlsanity.com/paper/2502.19180/) (2025)
- [How to Explain Neural Networks: an Approximation Perspective](https://mlsanity.com/paper/2105.07831/) (2021)
- [Exploring Hidden Semantics in Neural Networks with Symbolic Regression](https://mlsanity.com/paper/2204.10529/) (2022)
- [Improving classifier decision boundaries using nearest neighbors](https://mlsanity.com/paper/2310.03927/) (2023)
- [Decision Potential Surface: A Theoretical and Practical Approximation of Large Language Model Decision Boundary](https://mlsanity.com/paper/2510.03271/) (2025)

## Abstract (arXiv)

While deep learning models and techniques have achieved great empirical success, our understanding of the source of success in many aspects remains very limited. In an attempt to bridge the gap, we investigate the decision boundary of a production deep learning architecture with weak assumptions on both the training data and the model. We demonstrate, both theoretically and empirically, that the last weight layer of a neural network converges to a linear SVM trained on the output of the last hidden layer, for both the binary case and the multi-class case with the commonly used cross-entropy loss. Furthermore, we show empirically that training a neural network as a whole, instead of only fine-tuning the last weight layer, may result in better bias constant for the last weight layer, which is important for generalization. In addition to facilitating the understanding of deep learning, our result can be helpful for solving a broad range of practical problems of deep learning, such as catastrophic forgetting and adversarial attacking. The experiment codes are available at https://github.com/lykaust15/NN_decision_boundary