# Decision boundaries and convex hulls in the feature space that deep learning functions learn from images

arXiv: https://arxiv.org/abs/2202.04052 · HTML: https://mlsanity.com/paper/2202.04052/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 5 Feb 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Are adversarial examples inevitable?](https://mlsanity.com/paper/1809.02104/) (2018)
- [Large Margin Deep Networks for Classification](https://mlsanity.com/paper/1803.05598/) (2018)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [Learning in High Dimension Always Amounts to Extrapolation](https://mlsanity.com/paper/2110.09485/) (2021)
- [The Modern Mathematics of Deep Learning](https://mlsanity.com/paper/2105.04026/) (2021)
- [Cold Case: The Lost MNIST Digits](https://mlsanity.com/paper/1905.10498/) (2019)
- [Mad Max: Affine Spline Insights into Deep Learning](https://mlsanity.com/paper/1805.06576/) (2018)
- [Towards Learning Convolutions from Scratch](https://mlsanity.com/paper/2007.13657/) (2020)
- [Separation and Concentration in Deep Networks](https://mlsanity.com/paper/2012.10424/) (2020)
- [Deep Network Classification by Scattering and Homotopy Dictionary Learning](https://mlsanity.com/paper/1910.03561/) (2019)
- [Deep Learning Generalization and the Convex Hull of Training Sets](https://mlsanity.com/paper/2101.09849/) (2021)
- [A Sketching Method for Finding the Closest Point on a Convex Hull](https://mlsanity.com/paper/2102.10502/) (2021)
- [Image interpolation using Shearlet based iterative refinement](https://mlsanity.com/paper/1308.1126/) (2013)

## Built on by (most cited first, 5 of 5)

- [Reverse engineering adversarial attacks with fingerprints from adversarial examples](https://mlsanity.com/paper/2301.13869/) (2023)
- [SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation](https://mlsanity.com/paper/2511.10370/) (2025)
- [Scalable Attribution of Adversarial Attacks via Multi-Task Learning](https://mlsanity.com/paper/2302.14059/) (2023)
- [Over-parameterization: A Necessary Condition for Models that Extrapolate](https://mlsanity.com/paper/2203.10447/) (2022)
- [An Ambiguity Measure for Recognizing the Unknowns in Deep Learning](https://mlsanity.com/paper/2312.06077/) (2023)

## Abstract (arXiv)

The success of deep neural networks in image classification and learning can be partly attributed to the features they extract from images. It is often speculated about the properties of a low-dimensional manifold that models extract and learn from images. However, there is not sufficient understanding about this low-dimensional space based on theory or empirical evidence. For image classification models, their last hidden layer is the one where images of each class is separated from other classes and it also has the least number of features. Here, we develop methods and formulations to study that feature space for any model. We study the partitioning of the domain in feature space, identify regions guaranteed to have certain classifications, and investigate its implications for the pixel space. We observe that geometric arrangements of decision boundaries in feature space is significantly different compared to pixel space, providing insights about adversarial vulnerabilities, image morphing, extrapolation, ambiguity in classification, and the mathematical understanding of image classification models.