# Building a Regular Decision Boundary with Deep Networks

arXiv: https://arxiv.org/abs/1703.01775 · HTML: https://mlsanity.com/paper/1703.01775/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2017.

## Built on by (most cited first, 11 of 11)

- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Probing Representation Forgetting in Supervised and Unsupervised Continual Learning](https://mlsanity.com/paper/2203.13381/) (2022)
- [Can Forward Gradient Match Backpropagation?](https://mlsanity.com/paper/2306.06968/) (2023)
- [Conditional Mutual Information Constrained Deep Learning for Classification](https://mlsanity.com/paper/2309.09123/) (2023)
- [Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning](https://mlsanity.com/paper/2106.06401/) (2021)
- [Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation](https://mlsanity.com/paper/2402.04958/) (2024)
- [Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers](https://mlsanity.com/paper/2601.02543/) (2026)
- [Guidelines For The Choice Of The Baseline in XAI Attribution Methods](https://mlsanity.com/paper/2503.19813/) (2025)
- [Interferometric Graph Transform for Community Labeling](https://mlsanity.com/paper/2106.05875/) (2021)

## Abstract (arXiv)

In this work, we build a generic architecture of Convolutional Neural Networks to discover empirical properties of neural networks. Our first contribution is to introduce a state-of-the-art framework that depends upon few hyper parameters and to study the network when we vary them. It has no max pooling, no biases, only 13 layers, is purely convolutional and yields up to 95.4% and 79.6% accuracy respectively on CIFAR10 and CIFAR100. We show that the nonlinearity of a deep network does not need to be continuous, non expansive or point-wise, to achieve good performance. We show that increasing the width of our network permits being competitive with very deep networks. Our second contribution is an analysis of the contraction and separation properties of this network. Indeed, a 1-nearest neighbor classifier applied on deep features progressively improves with depth, which indicates that the representation is progressively more regular. Besides, we defined and analyzed local support vectors that separate classes locally. All our experiments are reproducible and code is available online, based on TensorFlow.