mlsanityData to 14 Oct 2026

Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compression

arXiv 2305.09896 ↗First posted 17 May 2023cs.LG math.OCcited by 9

Not on the map. Why?

Abstract (arXiv)

Achieving communication efficiency in decentralized machine learning has been attracting significant attention, with communication compression recognized as an effective technique in algorithm design. This paper takes a first step to understand the role of gradient clipping, a popular strategy in practice, in decentralized nonconvex optimization with communication compression. We propose PORTER, which considers two variants of gradient clipping added before or after taking a mini-batch of stochastic gradients, where the former variant PORTER-DP allows local differential privacy analysis with additional Gaussian perturbation, and the latter variant PORTER-GC helps to stabilize training. We develop a novel analysis framework that establishes their convergence guarantees without assuming the stringent bounded gradient assumption. To the best of our knowledge, our work provides the first convergence analysis for decentralized nonconvex optimization with gradient clipping and communication compression, highlighting the trade-offs between convergence rate, compression ratio, network connectivity, and privacy.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 47 of its 83 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

47 of 83 references.

References · 47

  1. 2016Deep Learning with Differential Privacycited by 2,728
  2. 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 903
  3. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  4. 2018Sparsified SGD with Memorycited by 376
  5. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  6. 2018The Convergence of Sparsified Gradient Methodscited by 229
  7. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  8. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
  9. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  10. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 150
  11. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  12. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
35 more
  1. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  2. 2017Network Topology and Communication-Computation Tradeoffs in Decentralized Optimizationcited by 122
  3. 2019Distributed Learning with Compressed Gradient Differencescited by 112
  4. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  5. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  6. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  7. 2018Communication Compression for Decentralized Trainingcited by 88
  8. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  9. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  10. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  11. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  12. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  13. 2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
  14. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  15. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  16. 2015Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithmscited by 31
  17. 2020SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimizationcited by 31
  18. 2021Dynamic Differential-Privacy Preserving SGDcited by 30
  19. 2020A general framework for decentralized optimization with first-order methodscited by 28
  20. 2019Efficient Privacy-Preserving Stochastic Nonconvex Optimizationcited by 27
  21. 2020Quantized Decentralized Stochastic Learning over Directed Graphscited by 24
  22. 2021CANITA: Faster Rates for Distributed Convex Optimization with Communication Compressioncited by 23
  23. 2019textttDeepSqueeze: Decentralization Meets Error-Compensated Compressioncited by 22
  24. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  25. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  26. 2020Fast decentralized non-convex finite-sum optimization with recursive variance reductioncited by 17
  27. 2021DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimizationcited by 16
  28. 2020A fast randomized incremental gradient method for decentralized non-convex optimizationcited by 14
  29. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  30. 2023Compressed Decentralized Proximal Stochastic Gradient Method for Nonconvex Composite Problems with Heterogeneous Datacited by 11
  31. 2022Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topologycited by 10
  32. 2023DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learningcited by 10
  33. 2023CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergencecited by 9
  34. 2020Scaling-up Distributed Processing of Data Streams for Machine Learningcited by 6
  35. 2023A Linearly Convergent Robust Compressed Push-Pull Method for Decentralized Optimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 3; 2026: 1.

9 papers.

  1. 2024The Privacy Power of Correlated Noise in Decentralized Learningcited by 18
  2. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  3. 2023Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compressioncited by 4
  4. 2024Communication-efficient Vertical Federated Learning via Compressed Error Feedbackcited by 3
  5. 2023Communication-Efficient Federated Optimization over Semi-Decentralized Networkscited by 2
  6. 2023Sparse Training for Federated Learning with Regularized Error Correctioncited by 2
  7. 2026Communication-Efficient Distributed Learning with Differential Privacycited by 1
  8. 2025Dyn-D2P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guaranteecited by 0
  9. 2025Balancing Utility and Privacy: Dynamically Private SGD with Random Projectioncited by 0