mlsanityData to 13 Oct 2026

Stochastic Rounding Increases Small Singular Values

arXiv 2606.00312 ↗First posted 29 May 2026math.NA cs.LG cs.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications spanning numerical analysis and modern machine learning systems. Recent work has shown that SR acts as an implicit regularizer by increasing the smallest singular value of extremely tall-and-thin (or, symmetrically, short-and-fat) matrices. In this work, we substantially sharpen and extend this understanding in two directions. First, we show that the regularization effect of SR is not restricted to extreme aspect ratio regimes: it persists for matrices with constant aspect ratio. Second, we demonstrate that SR does not merely regularize the smallest singular value, but instead lifts entire clusters of singular values at the tail of the spectrum. Together, these results provide a more general characterization of stochastic rounding as a spectral regularizer, revealing that its effects extend beyond extremal aspect ratios and act on a broader portion of the singular value spectrum.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

24 of 45 references.

References · 24

  1. 2017Mixed Precision Trainingcited by 1,147
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 814
  3. 2015Deep Learning with Limited Numerical Precisioncited by 372
  4. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 213
  6. 2022FP8 Formats for Deep Learningcited by 192
  7. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  8. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  9. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  10. 2025Training LLMs with MXFP4cited by 36
  11. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  12. 2025Recipes for Pre-training LLMs with MXFP8cited by 21
12 more
  1. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  2. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  3. 2025Effective Quantization of Muon Optimizer Statescited by 10
  4. 2024A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Trainingcited by 6
  5. 2024Direct Quantized Training of Language Models with Stochastic Roundingcited by 6
  6. 2025LOTION: Smoothing the Optimization Landscape for Quantized Trainingcited by 5
  7. 2023On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequalitycited by 4
  8. 2025Spectral Perturbation Bounds for Low-Rank Approximation with Applications to Privacycited by 3
  9. 2024Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matricescited by 2
  10. 2024Stochastic Rounding 2.0, with a View towards Complexity Analysiscited by 2
  11. 2024Error Analysis of Sum-Product Algorithms under Stochastic Roundingcited by 2
  12. 2026Eigenvalue stability and new perturbation bounds for the extremal eigenvalues of a matrixcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
  2. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantization
  3. 2025Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
  4. 2026Product-Aware Deterministic Rounding for Quantized Matrix Multiplication
  5. 2026LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment
  6. 2025pychop: Emulating Low-Precision Arithmetic in Numerical Methods and Neural Networks
  7. 2026Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition
  8. 2024Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models