# Statistical Inference for Stochastic Gradient Descent: Beyond Finite Variance

arXiv: https://arxiv.org/abs/2605.26000 · HTML: https://mlsanity.com/paper/2605.26000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 21 of 21)

- [A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks](https://mlsanity.com/paper/1901.06053/) (2019)
- [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails](https://mlsanity.com/paper/2106.14343/) (2021)
- [Statistical Inference for Model Parameters in Stochastic Gradient Descent](https://mlsanity.com/paper/1610.08637/) (2016)
- [Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks](https://mlsanity.com/paper/2006.09313/) (2020)
- [Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks](https://mlsanity.com/paper/2106.03795/) (2021)
- [On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks](https://mlsanity.com/paper/1912.00018/) (2019)
- [Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance](https://mlsanity.com/paper/2102.10346/) (2021)
- [Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling](https://mlsanity.com/paper/2106.03156/) (2021)
- [Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation](https://mlsanity.com/paper/2303.12277/) (2023)
- [Multiplicative noise and heavy tails in stochastic optimization](https://mlsanity.com/paper/2006.06293/) (2020)
- [Optimal variance-reduced stochastic approximation in Banach spaces](https://mlsanity.com/paper/2201.08518/) (2022)
- [Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise](https://mlsanity.com/paper/2102.04297/) (2021)
- [On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Means](https://mlsanity.com/paper/1911.01483/) (2019)
- [Online Bootstrap Inference with Nonconvex Stochastic Gradient Descent Estimator](https://mlsanity.com/paper/2306.02205/) (2023)
- [High Confidence Level Inference is Almost Free using Parallel Stochastic Optimization](https://mlsanity.com/paper/2401.09346/) (2024)
- [Emergence of heavy tails in homogenized stochastic gradient descent](https://mlsanity.com/paper/2402.01382/) (2024)
- [Limit Theorems for Stochastic Gradient Descent with Infinite Variance](https://mlsanity.com/paper/2410.16340/) (2024)
- [Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent](https://mlsanity.com/paper/2407.02322/) (2024)
- [Analysing heavy-tail properties of Stochastic Gradient Descent by means of Stochastic Recurrence Equations](https://mlsanity.com/paper/2403.13868/) (2024)

## Built on by (most cited first, 2 of 2)

- [Statistical Inference on Gradient Flows](https://mlsanity.com/paper/2606.01257/) (2026)
- [Geometric Moment Contraction for Stochastic Nesterov Acceleration](https://mlsanity.com/paper/2609.31303/) (2026)

## Abstract (arXiv)

Stochastic gradient descent (SGD) is foundational to large-scale statistical learning and stochastic optimization. However, in some modern statistical learning problems, stochastic gradients can exhibit infinite-variance behavior. Consequently, classical inference methods for SGD that rely on a finite-variance assumption break down. We develop a model-agnostic methodology for constructing confidence regions from SGD iterates in both the finite- and infinite-variance regimes. We first show that Polyak--Ruppert averaging has an asymptotic directional scale no larger than that of the fastest-rate final iterate, analogous to its lower asymptotic variance in the finite-variance setting. Accordingly, we focus our inference methodology on the Polyak--Ruppert averaged estimator. Specifically, we establish a joint central limit theorem for this estimator and an empirical second-moment normalizer from the same iterates. This joint limit yields a self-normalized statistic in which the leading tail-dependent scaling terms cancel. We then use subsampling to estimate the relevant quantiles, avoiding explicit estimation of nuisance parameters including tail indices, slowly varying functions, or stable-law parameters. The resulting confidence regions are straightforward to implement and asymptotically valid in both the finite- and infinite-variance regimes. Empirical studies show reliable coverage in various settings, supporting the proposed method as a practical tool for uncertainty quantification in stochastic optimization.