Robust learning with anytime-guaranteed feedback
Not on the map. Why?
Abstract (arXiv)
Under data distributions which may be heavy-tailed, many stochastic gradient-based learning algorithms are driven by feedback queried at points with almost no performance guarantees on their own. Here we explore a modified "anytime online-to-batch" mechanism which for smooth objectives admits high-probability error bounds while requiring only lower-order moment bounds on the stochastic gradients. Using this conversion, we can derive a wide variety of "anytime robust" procedures, for which the task of performance analysis can be effectively reduced to regret control, meaning that existing regret bounds (for the bounded gradient case) can be robustified and leveraged in a straightforward manner. As a direct takeaway, we obtain an easily implemented stochastic gradient-based algorithm for which all queried points formally enjoy sub-Gaussian error bounds, and in practice show noteworthy gains on real-world data applications.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 35 references that are in the map. First posted on arXiv 24 May 2021.
Builds on
10 of 35 references.
References · 10
- 2019Introduction to Online Convex Optimizationcited by 803
- 2014Convex Optimization: Algorithms and Complexitycited by 575
- 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
- 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
- 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 95
- 2018Robust Estimation via Robust Gradient Estimationcited by 76
- 2016Geometric Median in Nearly Linear Timecited by 39
- 2014Empirical risk minimization for heavy-tailed lossescited by 37
- 2019Algorithms of Robust Stochastic Optimization Based on Mirror Descent Methodcited by 32
- 2019From low probability to high confidence in stochastic convex optimizationcited by 1
Built on by
Not cited yet.