# (Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number

arXiv: https://arxiv.org/abs/2410.00169 · HTML: https://mlsanity.com/paper/2410.00169/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 22 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 3 of 3)

- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [Stabilizing Invertible Neural Networks Using Mixture Models](https://mlsanity.com/paper/2009.02994/) (2020)

## Built on by (most cited first, 5 of 5)

- [Model Immunization from a Condition Number Perspective](https://mlsanity.com/paper/2505.23760/) (2025)
- [Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks](https://mlsanity.com/paper/2505.08022/) (2025)
- [ISAC: An Invertible and Stable Auditory Filter Bank with Customizable Kernels for ML Integration](https://mlsanity.com/paper/2505.07709/) (2025)
- [Aliasing in Convnets: A Frame-Theoretic Perspective](https://mlsanity.com/paper/2507.06152/) (2025)
- [Demystifying Lipschitz verification: positive matrices, negative results](https://mlsanity.com/paper/2603.28113/) (2026)

## Abstract (arXiv)

Maintaining numerical stability in machine learning models is crucial for their reliability and performance. One approach to maintain stability of a network layer is to integrate the condition number of the weight matrix as a regularizing term into the optimization algorithm. However, due to its discontinuous nature and lack of differentiability the condition number is not suitable for a gradient descent approach. This paper introduces a novel regularizer that is provably differentiable almost everywhere and promotes matrices with low condition numbers. In particular, we derive a formula for the gradient of this regularizer which can be easily implemented and integrated into existing optimization algorithms. We show the advantages of this approach for noisy classification and denoising of MNIST images.