mlsanityData to 7 Oct 2026

An Empirical Study on Compressed Decentralized Stochastic Gradient Algorithms with Overparameterized Models

arXiv 2110.04523 ↗First posted 9 Oct 2021math.OC cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper considers decentralized optimization with application to machine learning on graphs. The growing size of neural network (NN) models has motivated prior works on decentralized stochastic gradient algorithms to incorporate communication compression. On the other hand, recent works have demonstrated the favorable convergence and generalization properties of overparameterized NNs. In this work, we present an empirical analysis on the performance of compressed decentralized stochastic gradient (DSG) algorithms with overparameterized NNs. Through simulations on an MPI network environment, we observe that the convergence rates of popular compressed DSG algorithms are robust to the size of NNs. Our findings suggest a gap between theories and practice of the compressed DSG algorithms in the existing literature.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

23 of 32 references.

References · 23

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  5. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  6. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  8. 2018The Convergence of Sparsified Gradient Methodscited by 228
  9. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  10. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  11. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  12. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
11 more
  1. 2018D2: Decentralized Training over Decentralized Datacited by 140
  2. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  3. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  4. 2018Communication Compression for Decentralized Trainingcited by 87
  5. 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
  6. 2020An improved convergence analysis for decentralized online stochastic non-convex optimizationcited by 53
  7. 2016On Nonconvex Decentralized Gradient Descentcited by 42
  8. 2020A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!cited by 41
  9. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  10. 2018Learning Two Layer Rectified Neural Networks in Polynomial Timecited by 31
  11. 2019Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approachcited by 11

Built on by

Not cited yet.