Data to 14 Oct 2026

Gradient Sparsification for Communication-Efficient Distributed Optimization

arXiv 1710.09854 ↗First posted 26 Oct 2017cs.LG cs.NA math.NAcited by 201

Not on the map. Why?

Abstract (arXiv)

Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost we propose a convex optimization formulation to minimize the coding length of stochastic gradients. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for approximate solution, with theoretical guaranteed for sparseness. Experiments on ℓ2 regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

Key facts. Cited by 201 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 59; 2022: 46; 2023: 31; 2024: 31; 2025: 18; 2026: 15.

201 papers.

Sort:

By area:

2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
2022Federated Learning via Inexact ADMMcited by 44
2021Rethinking gradient sparsification as total error minimizationcited by 41
2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
2023A Guide Through the Zoo of Biased SGDcited by 40
2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 37

38 more

2021Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraintscited by 36
2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
2021DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregationcited by 31
2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
2022Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applicationscited by 27
2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 26
2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
2021An Information-Theoretic Justification for Model Pruningcited by 25
2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 25
20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
2021CANITA: Faster Rates for Distributed Convex Optimization with Communication Compressioncited by 23
2023Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimationcited by 22
2021Preserved central model for faster bidirectional compression in distributed settingscited by 21
2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 21
2021Learned Gradient Compression for Distributed Deep Learningcited by 20
2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
2021Communication-Efficient Federated Learning via Quantized Compressed Sensingcited by 18
2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
2021Distributed Learning Systems with First-order Methodscited by 17
2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimizationcited by 17
2021Truly Sparse Neural Networks at Scalecited by 16
2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
2023FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropoutcited by 16
2021FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learningcited by 15
2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 15
2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
2022Federated Few-Shot Learning for Mobile NLPcited by 15
2021MURANA: A Generic Framework for Stochastic Variance-Reduced Optimizationcited by 14
2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
2022Communication-Efficient Federated Learning for Heterogeneous Edge Devices Based on Adaptive Gradient Quantizationcited by 14

50 most cited of 201 All 201 (JSON)