mlsanityData to 14 Oct 2026

Deep learning with Elastic Averaging SGD

arXiv 1412.6651 ↗First posted 20 Dec 2014cs.LG stat.MLcited by 90

Not on the map. Why?

Abstract (arXiv)

We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 29; 2022: 13; 2023: 16; 2024: 14; 2025: 11; 2026: 6.

90 papers.

Sort:

By area:

  1. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  2. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  3. 2022Federated Learning via Inexact ADMMcited by 44
  4. 2021Anarchic Federated Learningcited by 42
  5. 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
  6. 2021Communication Efficiency in Federated Learning: Achievements and Challengescited by 40
  7. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  8. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  9. 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
  10. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  11. 2021Distributed Learning Systems with First-order Methodscited by 17
  12. 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
38 more
  1. 2023On the Role of Server Momentum in Federated Learningcited by 16
  2. 2022Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updatescited by 15
  3. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  4. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 15
  5. 2026Demystifying Data-Driven Probabilistic Medium-Range Weather Forecastingcited by 15
  6. 2021Asynchronous Federated Learning for Sensor Data with Concept Driftcited by 14
  7. 2021Communication-Efficient ADMM-based Federated Learningcited by 14
  8. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  9. 2021Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settingscited by 11
  10. 2021Federated Few-Shot Learning with Adversarial Learningcited by 11
  11. 2021GIST: Distributed Training for Large-Scale Graph Convolutional Networkscited by 10
  12. 2021JUWELS Booster -- A Supercomputer for Large-Scale AI Researchcited by 10
  13. 2021Large-scale Building Height Retrieval from Single SAR Imagery based on Bounding Box Regression Networkscited by 10
  14. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  15. 2025FeDaL: Federated Dataset Learning for General Time Series Foundation Modelscited by 10
  16. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 9
  17. 2021Efficient and Reliable Overlay Networks for Decentralized Federated Learningcited by 9
  18. 2023Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimalitycited by 8
  19. 2023Learning Rate Schedules in the Presence of Distribution Shiftcited by 7
  20. 2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
  21. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  22. 2023An effective theory of collective deep learningcited by 7
  23. 2025HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmarkcited by 7
  24. 2021Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learningcited by 6
  25. 2022Personalized Federated Learning with Communication Compressioncited by 6
  26. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
  27. 2021The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communicationcited by 5
  28. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
  29. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  30. 2021Partitioning sparse deep neural networks for scalable training and inferencecited by 4
  31. 2021Better Training using Weight-Constrained Stochastic Dynamicscited by 4
  32. 2022Enabling All In-Edge Deep Learning: A Literature Reviewcited by 4
  33. 2023DropCompute: simple and more robust distributed synchronous training via compute variance reductioncited by 4
  34. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  35. 2024GRAWA: Gradient-based Weighted Averaging for Distributed Training of Deep Learning Modelscited by 4
  36. 2024New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecturecited by 4
  37. 2020Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcited by 3
  38. 2021Distributed stochastic optimization with large delayscited by 3

50 most cited of 90 All 90 (JSON)