The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights
Not on the map. Why?
Abstract (arXiv)
We analyze gradient descent with randomly weighted data points in a linear regression model, under a generic weighting distribution. This includes various forms of stochastic gradient descent, importance sampling, but also extends to weighting distributions with arbitrary continuous values, thereby providing a unified framework to analyze the impact of various kinds of noise on the training trajectory. We characterize the implicit regularization induced through the random weighting, connect it with weighted linear regression, and derive non-asymptotic bounds for convergence in first and second moments. Leveraging geometric moment contraction, we also investigate the stationary distribution induced by the added noise. Based on these results, we discuss how specific choices of weighting distribution influence both the underlying optimization problem and statistical properties of the resulting estimator, as well as some examples for which weightings that lead to fast convergence cause bad statistical performance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 11 Dec 2025.
Builds on
28 of 50 references.
References · 28
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
- 2021Deep learning: a statistical viewpointcited by 240
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
- 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 133
- 2018Stochastic subgradient method converges on tame functionscited by 124
- 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
- 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
- 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
16 more
- 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
- 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 48
- 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
- 2016Importance Sampling for Minibatchescited by 36
- 2014Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squarescited by 34
- 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
- 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
- 2021Convergence of gradient descent for learning linear neural networkscited by 22
- 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
- 2024What is the long-run distribution of stochastic gradient descent? A large deviations analysiscited by 17
- 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
- 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
- 2024Singular-limit analysis of gradient descent with noise injectioncited by 7
- 2024Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Modelscited by 7
- 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
- 2025Decorrelated feature importance from local sample weightingcited by 1
Built on by
Not cited yet.