# Efficient Decentralized Deep Learning by Dynamic Model Averaging

arXiv: https://arxiv.org/abs/1807.03210 · HTML: https://mlsanity.com/paper/1807.03210/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 30 of 30)

- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Robustness to Unbounded Smoothness of Generalized SignSGD](https://mlsanity.com/paper/2208.11195/) (2022)
- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [Gradient-Leakage Resilient Federated Learning](https://mlsanity.com/paper/2107.01154/) (2021)
- [Communication Efficiency in Federated Learning: Achievements and Challenges](https://mlsanity.com/paper/2107.10996/) (2021)
- [FEDZIP: A Compression Framework for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.01593/) (2021)
- [Layer-wise Linear Mode Connectivity](https://mlsanity.com/paper/2307.06966/) (2023)
- [Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity](https://mlsanity.com/paper/2102.04635/) (2021)
- [Trade-offs of Local SGD at Scale: An Empirical Study](https://mlsanity.com/paper/2110.08133/) (2021)
- [Semi-Decentralized Federated Learning with Collaborative Relaying](https://mlsanity.com/paper/2205.10998/) (2022)
- [Federated Learning from Small Datasets](https://mlsanity.com/paper/2110.03469/) (2021)
- [Large-Scale Deep Learning Optimizations: A Comprehensive Survey](https://mlsanity.com/paper/2111.00856/) (2021)
- [Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies](https://mlsanity.com/paper/2109.03999/) (2021)
- [Experimenting with Normalization Layers in Federated Learning on non-IID scenarios](https://mlsanity.com/paper/2303.10630/) (2023)
- [A Re-Parameterized Vision Transformer (ReVT) for Domain-Generalized Semantic Segmentation](https://mlsanity.com/paper/2308.13331/) (2023)
- [Robust Federated Learning with Connectivity Failures: A Semi-Decentralized Framework with Collaborative Relaying](https://mlsanity.com/paper/2202.11850/) (2022)
- [Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning](https://mlsanity.com/paper/2310.05696/) (2023)
- [Version age-based client scheduling policy for federated learning](https://mlsanity.com/paper/2402.05407/) (2024)
- [Attentive Federated Learning for Concept Drift in Distributed 5G Edge Networks](https://mlsanity.com/paper/2111.07457/) (2021)
- [Tailored Uncertainty Estimation for Deep Learning Systems](https://mlsanity.com/paper/2204.13963/) (2022)
- [UniFed: A Unified Framework for Federated Learning on Non-IID Image Features](https://mlsanity.com/paper/2110.09974/) (2021)
- [Towards Reliable Participation in UAV-Enabled Federated Edge Learning on Non-IID Data](https://mlsanity.com/paper/2312.10411/) (2023)
- [Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2503.11146/) (2025)
- [A V2X-based Privacy Preserving Federated Measuring and Learning System](https://mlsanity.com/paper/2401.13848/) (2024)
- [A Mallows-like Criterion for Anomaly Detection with Random Forest Implementation](https://mlsanity.com/paper/2405.18932/) (2024)
- [Learning with Limited Shared Information in Multi-agent Multi-armed Bandit](https://mlsanity.com/paper/2502.15338/) (2025)
- [Topology-Aware Knowledge Propagation in Decentralized Learning](https://mlsanity.com/paper/2505.11760/) (2025)
- [Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs](https://mlsanity.com/paper/2605.11857/) (2026)
- [AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization](https://mlsanity.com/paper/2609.36662/) (2026)

## Abstract (arXiv)

We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared to periodically communicating state-of-the-art approaches. Moreover, we derive a communication bound that scales well with the hardness of the serialized learning problem. The reduction in communication comes at almost no cost, as the predictive performance remains virtually unchanged. Indeed, the proposed protocol retains loss bounds of periodically averaging schemes. An extensive empirical evaluation validates major improvement of the trade-off between model performance and communication which could be beneficial for numerous decentralized learning applications, such as autonomous driving, or voice recognition and image classification on mobile phones.