# Accumulated Gradient Normalization

arXiv: https://arxiv.org/abs/1710.02368 · HTML: https://mlsanity.com/paper/1710.02368/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2017.

## Built on by (most cited first, 10 of 10)

- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [Improving Mandarin Speech Recogntion with Block-augmented Transformer](https://mlsanity.com/paper/2207.11697/) (2022)
- [Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training](https://mlsanity.com/paper/2407.15892/) (2024)
- [Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2406.02733/) (2024)
- [A Distributional Perspective on Word Learning in Neural Language Models](https://mlsanity.com/paper/2502.05892/) (2025)
- [Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique](https://mlsanity.com/paper/2308.00197/) (2023)
- [InDistill: Information flow-preserving knowledge distillation for model compression](https://mlsanity.com/paper/2205.10003/) (2022)
- [Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets](https://mlsanity.com/paper/2305.11625/) (2023)
- [An Iterative Framework for Generative Backmapping of Coarse Grained Proteins](https://mlsanity.com/paper/2505.18082/) (2025)
- [AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training](https://mlsanity.com/paper/2605.29664/) (2026)

## Abstract (arXiv)

This work addresses the instability in asynchronous data parallel optimization. It does so by introducing a novel distributed optimizer which is able to efficiently optimize a centralized model under communication constraints. The optimizer achieves this by pushing a normalized sequence of first-order gradients to a parameter server. This implies that the magnitude of a worker delta is smaller compared to an accumulated gradient, and provides a better direction towards a minimum compared to first-order gradients, which in turn also forces possible implicit momentum fluctuations to be more aligned since we make the assumption that all workers contribute towards a single minima. As a result, our approach mitigates the parameter staleness problem more effectively since staleness in asynchrony induces (implicit) momentum, and achieves a better convergence rate compared to other optimizers such as asynchronous EASGD and DynSGD, which we show empirically.