barGmst:An Unbiased Stratified Statistic and a Fast Gradient Optimization Algorithm Based on It
Not on the map. Why?
Abstract (arXiv)
-The fluctuation effect of gradient expectation and variance caused by parameter update between consecutive iterations is neglected or confusing by current mainstream gradient optimization algorithms. The work in this paper remedy this issue by introducing a novel unbiased stratified statistic barGmst , a sufficient condition of fast convergence for barGmst also is established. A novel algorithm named MSSG designed based on barGmst outperforms other sgd-like algorithms. Theoretical conclusions and experimental evidence strongly suggest to employ MSSG when training deep model.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 26 references that are in the map. First posted on arXiv 7 Oct 2021.
Builds on
7 of 26 references.
References · 7
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2019On the Convergence of Adam and Beyondcited by 841
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
- 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
- 2018Large-Scale Visual Speech Recognitioncited by 57
- 2017A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexitycited by 3
Built on by
Not cited yet.