# Numerical stability analysis of large language models

arXiv: https://arxiv.org/abs/2503.10251 · HTML: https://mlsanity.com/paper/2503.10251/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2025.

## Built on by (most cited first, 9 of 9)

- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026)
- [Mixed precision accumulation for neural network inference guided by componentwise forward error analysis](https://mlsanity.com/paper/2503.15568/) (2025)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models](https://mlsanity.com/paper/2601.21623/) (2026)
- [A Note on Non-Composability of Layerwise Approximate Verification for Neural Inference](https://mlsanity.com/paper/2602.15756/) (2026)
- [Automated Numerical Stability Analysis of Deep Learning Operators](https://mlsanity.com/paper/2607.25494/) (2026)
- [LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators](https://mlsanity.com/paper/2609.39361/) (2026)
- [Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference](https://mlsanity.com/paper/2610.14277/) (2026)

## Abstract (arXiv)

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.