# DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learning

arXiv: https://arxiv.org/abs/2202.03493 · HTML: https://mlsanity.com/paper/2202.03493/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 29 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 5 of 5)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Adversarial Machine Learning -- Industry Perspectives](https://mlsanity.com/paper/2002.05646/) (2020)
- [Stochastic gradient variational Bayes for gamma approximating distributions](https://mlsanity.com/paper/1509.01631/) (2015)

## Built on by (most cited first, 11 of 11)

- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [MixQuant: Mixed Precision Quantization with a Bit-width Optimization Search](https://mlsanity.com/paper/2309.17341/) (2023)
- [Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects](https://mlsanity.com/paper/2302.06086/) (2023)
- [Fuzzy PyTorch: Rapid Numerical Variability Evaluation for Deep Learning Models](https://mlsanity.com/paper/2605.25991/) (2026)
- [Trusted Weights, Treacherous Optimizations? Optimization-Triggered Backdoor Attacks on LLMs](https://mlsanity.com/paper/2605.20641/) (2026)
- [A Retrospective on ICSE 2022](https://mlsanity.com/paper/2207.12578/) (2022)
- [Numerical Stability of DeepGOPlus Inference](https://mlsanity.com/paper/2212.06361/) (2022)
- [SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization](https://mlsanity.com/paper/2603.29109/) (2026)
- [Demystifying the Silence of Correctness Bugs in PyTorch Compiler](https://mlsanity.com/paper/2604.08720/) (2026)

## Abstract (arXiv)

Deep learning (DL) has become an integral part of solutions to various important problems, which is why ensuring the quality of DL systems is essential. One of the challenges of achieving reliability and robustness of DL software is to ensure that algorithm implementations are numerically stable. DL algorithms require a large amount and a wide variety of numerical computations. A naive implementation of numerical computation can lead to errors that may result in incorrect or inaccurate learning and results. A numerical algorithm or a mathematical formula can have several implementations that are mathematically equivalent, but have different numerical stability properties. Designing numerically stable algorithm implementations is challenging, because it requires an interdisciplinary knowledge of software engineering, DL, and numerical analysis. In this paper, we study two mature DL libraries PyTorch and Tensorflow with the goal of identifying unstable numerical methods and their solutions. Specifically, we investigate which DL algorithms are numerically unstable and conduct an in-depth analysis of the root cause, manifestation, and patches to numerical instabilities. Based on these findings, we launch, the first database of numerical stability issues and solutions in DL. Our findings and provide future references to developers and tool builders to prevent, detect, localize and fix numerically unstable algorithm implementations. To demonstrate that, using it DeepStability we have located numerical stability issues in Tensorflow, and submitted a fix which has been accepted and merged in.