Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning
Not on the map. Why?
Abstract (arXiv)
Decentralized Multi-agent Learning (DML) enables collaborative model training while preserving data privacy. However, inherent heterogeneity in agents' resources (computation, communication, and task size) may lead to substantial variations in training time. This heterogeneity creates a bottleneck, lengthening the overall training time due to straggler effects and potentially wasting spare resources of faster agents. To minimize training time in heterogeneous environments, we present a Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learning (ComDML), which balances the workload among agents through a decentralized approach. Leveraging local-loss split training, ComDML enables parallel updates, where slower agents offload part of their workload to faster agents. To minimize the overall training time, ComDML optimizes the workload balancing by jointly considering the communication and computation capacities of agents, which hinges upon integer programming. A dynamic decentralized pairing scheduler is developed to efficiently pair agents and determine optimal offloading amounts. We prove that in ComDML, both slower and faster agents' models converge, for convex and non-convex functions. Furthermore, extensive experimental results on popular datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants, with large models such as ResNet-56 and ResNet-110, demonstrate that ComDML can significantly reduce the overall training time while maintaining model accuracy, compared to state-of-the-art methods. ComDML demonstrates robustness in heterogeneous environments, and privacy measures can be seamlessly integrated for enhanced data protection.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 1 May 2024.
Builds on
30 of 44 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2016Deep Learning with Differential Privacycited by 2,682
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
- 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
18 more
- 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
- 2020Threats to Federated Learning: A Surveycited by 171
- 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
- 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 115
- 2021Decentralized Federated Averagingcited by 114
- 2020TiFL: A Tier-based Federated Learning Systemcited by 95
- 2019Peer-to-peer Federated Learning on Graphscited by 85
- 2020A Blockchain-based Decentralized Federated Learning Framework with Committee Consensuscited by 83
- 2019Decoupled Greedy Learning of CNNscited by 76
- 2020NoPeek: Information leakage reduction to share activations in distributed deep learningcited by 66
- 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
- 2019On Safeguarding Privacy and Security in the Framework of Federated Learningcited by 55
- 2019BAFFLE : Blockchain Based Aggregator Free Federated Learningcited by 38
- 2021SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learningcited by 20
- 2023Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharingcited by 7
- 2023Speed Up Federated Learning in Heterogeneous Environment: A Dynamic Tiering Approachcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.