# Zero Stability Well Predicts Performance of Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2206.13100 · HTML: https://mlsanity.com/paper/2206.13100/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Exploring Randomly Wired Neural Networks for Image Recognition](https://mlsanity.com/paper/1904.01569/) (2019)
- [On Robustness of Neural Ordinary Differential Equations](https://mlsanity.com/paper/1910.05513/) (2019)
- [PolyNet: A Pursuit of Structural Diversity in Very Deep Networks](https://mlsanity.com/paper/1611.05725/) (2016)
- [IMEXnet: A Forward Stable Deep Neural Network](https://mlsanity.com/paper/1903.02639/) (2019)
- [Convolutional Neural Networks combined with Runge-Kutta Methods](https://mlsanity.com/paper/1802.08831/) (2018)
- [ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies](https://mlsanity.com/paper/1811.10745/) (2018)
- [Towards Robust ResNet: A Small Step but A Giant Leap](https://mlsanity.com/paper/1902.10887/) (2019)
- [Activated Gradients for Deep Neural Networks](https://mlsanity.com/paper/2107.04228/) (2021)
- [Forward Stability of ResNet and Its Variants](https://mlsanity.com/paper/1811.09885/) (2018)
- [Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2112.12707/) (2021)
- [Review: Ordinary Differential Equations For Deep Learning](https://mlsanity.com/paper/1911.00502/) (2019)

## Built on by (most cited first, 1 of 1)

- [A topological description of loss surfaces based on Betti Numbers](https://mlsanity.com/paper/2401.03824/) (2024)

## Abstract (arXiv)

The question of what kind of convolutional neural network (CNN) structure performs well is fascinating. In this work, we move toward the answer with one more step by connecting zero stability and model performance. Specifically, we found that if a discrete solver of an ordinary differential equation is zero stable, the CNN corresponding to that solver performs well. We first give the interpretation of zero stability in the context of deep learning and then investigate the performance of existing first- and second-order CNNs under different zero-stable circumstances. Based on the preliminary observation, we provide a higher-order discretization to construct CNNs and then propose a zero-stable network (ZeroSNet). To guarantee zero stability of the ZeroSNet, we first deduce a structure that meets consistency conditions and then give a zero stable region of a training-free parameter. By analyzing the roots of a characteristic equation, we theoretically obtain the optimal coefficients of feature maps. Empirically, we present our results from three aspects: We provide extensive empirical evidence of different depth on different datasets to show that the moduli of the characteristic equation's roots are the keys for the performance of CNNs that require historical features; Our experiments show that ZeroSNet outperforms existing CNNs which is based on high-order discretization; ZeroSNets show better robustness against noises on the input. The source code is available at urlhttps://github.com/LongJin-lab/ZeroSNet.