# Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices

arXiv: https://arxiv.org/abs/2403.12278 · HTML: https://mlsanity.com/paper/2403.12278/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 38 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 3 of 3)

- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)

## Built on by (most cited first, 2 of 2)

- [Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learning](https://mlsanity.com/paper/2406.11151/) (2024)
- [Stochastic Rounding Increases Small Singular Values](https://mlsanity.com/paper/2606.00312/) (2026)

## Abstract (arXiv)

Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness rounding of real matrices A with many more rows than columns. We provide novel theoretical evidence, supported by extensive experimental evaluation that, with high probability, the smallest singular value of a stochastically rounded matrix is well bounded away from zero -- regardless of how close A is to being rank deficient and even if A is rank-deficient. In other words, stochastic rounding textitimplicitly regularizes tall and skinny matrices A so that the rounded version has full column rank. Our proofs leverage powerful results in random matrix theory, and the idea that stochastic rounding errors do not concentrate in low-dimensional column spaces.