# Convergence Rate for the Last Iterate of Stochastic Gradient Descent Schemes

arXiv: https://arxiv.org/abs/2507.07281 · HTML: https://mlsanity.com/paper/2507.07281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 9 Jul 2025.

## Builds on (most cited first, 4 of 4)

- [Stochastic Heavy Ball](https://mlsanity.com/paper/1609.04228/) (2016)
- [New Convergence Aspects of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1811.12403/) (2018)
- [Almost sure convergence rates of stochastic gradient methods under gradient domination](https://mlsanity.com/paper/2405.13592/) (2024)
- [Convergence of Unregularized Online Learning Algorithms](https://mlsanity.com/paper/1708.02939/) (2017)

## Abstract (arXiv)

We study the convergence rate for the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) in the parametric setting when the objective function F is globally convex or non-convex whose gradient is γ-H"older. Using only discrete Gronwall's inequality without Robbins-Siegmund theorem, we recover results for both SGD and SHB: mins≤ t |nabla F(ws)|2 = o(tp-1) for non-convex objectives and F(wτ wedge t) - F_* = o(t2γ/(1+γ) · max(p-1,-2p+1)-ε) for β ∈ (0, 1), τ := inf  t > 0 : F(wt) = F_*, and mins ≤ t F(ws) - F_* = o(tp-1) for convex objectives F whose minimum is F_*. In addition, we proved that SHB with constant momentum parameter β ∈ (0, 1) attains a convergence rate of F(wt) - F_* = O(tmax(p-1,-2p+1) log2 fractδ) with probability at least 1-δ when F is convex and γ = 1 and step size αt = Theta(t-p) with p ∈ (frac12, 1).