# Quantized Adam with Error Feedback

arXiv: https://arxiv.org/abs/2004.14180 · HTML: https://mlsanity.com/paper/2004.14180/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 27 of 27)

- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [Communication-Efficient Adaptive Federated Learning](https://mlsanity.com/paper/2205.02719/) (2022)
- [FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy](https://mlsanity.com/paper/2302.10429/) (2023)
- [A Guide Through the Zoo of Biased SGD](https://mlsanity.com/paper/2305.16296/) (2023)
- [AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning](https://mlsanity.com/paper/2211.15055/) (2022)
- [Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration](https://mlsanity.com/paper/2101.05471/) (2021)
- [Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup](https://mlsanity.com/paper/2308.00522/) (2023)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [On Distributed Adaptive Optimization with Gradient Compression](https://mlsanity.com/paper/2205.05632/) (2022)
- [AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning](https://mlsanity.com/paper/2410.17881/) (2024)
- [Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning](https://mlsanity.com/paper/2111.01847/) (2021)
- [Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization](https://mlsanity.com/paper/2111.00705/) (2021)
- [Efficient-Adam: Communication-Efficient Distributed Adam](https://mlsanity.com/paper/2205.14473/) (2022)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Stochastic Rounding for LLM Training: Theory and Practice](https://mlsanity.com/paper/2502.20566/) (2025)
- [LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM](https://mlsanity.com/paper/2502.19571/) (2025)
- [Unified Scaling Laws for Compressed Representations](https://mlsanity.com/paper/2506.01863/) (2025)
- [CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training](https://mlsanity.com/paper/2510.18784/) (2025)
- [Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization](https://mlsanity.com/paper/2106.10022/) (2021)
- [FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data](https://mlsanity.com/paper/2309.09719/) (2023)
- [Communication-Efficient Distributed Learning with Local Immediate Error Compensation](https://mlsanity.com/paper/2402.11857/) (2024)
- [Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/2508.04950/) (2025)
- [Communication-Efficient Adam-Type Algorithms for Distributed Data Mining](https://mlsanity.com/paper/2210.07454/) (2022)
- [Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization](https://mlsanity.com/paper/2405.17932/) (2024)
- [Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis](https://mlsanity.com/paper/2411.06770/) (2024)
- [Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification](https://mlsanity.com/paper/2412.01195/) (2024)
- [LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication](https://mlsanity.com/paper/2604.09970/) (2026)

## Abstract (arXiv)

In this paper, we present a distributed variant of adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce the communication cost among the workers and server, we incorporate two types of quantization schemes, i.e., gradient quantization and weight quantization, into the proposed distributed Adam. Besides, to reduce the bias introduced by quantization operations, we propose an error-feedback technique to compensate for the quantized gradient. Theoretically, in the stochastic nonconvex setting, we show that the distributed adaptive gradient method with gradient quantization and error-feedback converges to the first-order stationary point, and that the distributed adaptive gradient method with weight quantization and error-feedback converges to the point related to the quantized level under both the single-worker and multi-worker modes. At last, we apply the proposed distributed adaptive gradient methods to train deep neural networks. Experimental results demonstrate the efficacy of our methods.