mlsanityData to 7 Oct 2026

Bias-Variance Trade-off for Clipped Stochastic First-Order Methods: From Bounded Variance to Infinite Mean

arXiv 2512.14686 ↗First posted 16 Dec 2025cs.LG cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic optimization is fundamental to modern machine learning. Recent research has extended the study of stochastic first-order methods (SFOMs) from light-tailed to heavy-tailed noise, which frequently arises in practice, with clipping emerging as a key technique for controlling heavy-tailed gradients. Extensive theoretical advances have further shown that the oracle complexity of SFOMs depends on the tail index α of the noise. Nonetheless, existing complexity results often cover only the case α ∈ (1,2], that is, the regime where the noise has a finite mean, while the complexity bounds tend to infinity as α approaches 1. This paper tackles the general case of noise with tail index α∈(0,2], covering regimes ranging from noise with bounded variance to noise with an infinite mean, where the latter case has been scarcely studied. Through a novel analysis of the bias-variance trade-off in gradient clipping, we show that when a symmetry measure of the noise tail is controlled, clipped SFOMs achieve improved complexity guarantees in the presence of heavy-tailed noise for any tail index α ∈ (0,2]. Our analysis of the bias-variance trade-off not only yields new unified complexity guarantees for clipped SFOMs across this full range of tail indices, but is also straightforward to apply and can be combined with classical analyses under light-tailed noise to establish oracle complexity guarantees under heavy-tailed noise. Finally, numerical experiments validate our theoretical findings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

24 of 46 references.

References · 24

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  4. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  5. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  6. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  7. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  8. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  9. 2023High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variancecited by 60
  10. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  11. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  12. 2019The Complexity of Making the Gradient Small in Stochastic Convex Optimizationcited by 33
12 more
  1. 2023Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problemscited by 21
  2. 2024Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Accelerationcited by 18
  3. 2022Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noisecited by 16
  4. 2025Can SGD Handle Heavy-Tailed Noise?cited by 15
  5. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 13
  6. 2025Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noisecited by 11
  7. 2024Non-convex Stochastic Composite Optimization with Polyak Momentumcited by 9
  8. 2024Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetrycited by 7
  9. 2022A single cut proximal bundle method for stochastic convex composite optimizationcited by 4
  10. 2024Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clippingcited by 3
  11. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3
  12. 2025Accelerated stochastic first-order method for convex optimization under heavy-tailed noisecited by 3

Built on by

Not cited yet.