Straggler-Resilient Differentially-Private Decentralized Learning
Not on the map. Why?
Abstract (arXiv)
We consider the straggler problem in decentralized learning over a logical ring while preserving user data privacy. Especially, we extend the recently proposed framework of differential privacy (DP) amplification by decentralization by Cyffers and Bellet to include overall training latency--comprising both computation and communication latency. Analytical results on both the convergence speed and the DP level are derived for both a skipping scheme (which ignores the stragglers after a timeout) and a baseline scheme that waits for each node to finish before the training continues. A trade-off between overall training latency, accuracy, and privacy, parameterized by the timeout of the skipping scheme, is identified and empirically validated for logistic regression on a real-world dataset and for image classification using the MNIST and CIFAR-10 datasets.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 6 Dec 2022.
Builds on
31 of 51 references.
References · 31
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2017Renyi Differential Privacycited by 643
- 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
- 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
- 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
- 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
- 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
- 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 163
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
19 more
- 2018Privacy Amplification by Iterationcited by 97
- 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
- 2015Speeding Up Distributed Machine Learning Using Codescited by 76
- 2018On Markov Chain Gradient Descentcited by 63
- 2020Decentralized gradient methods: does topology matter?cited by 35
- 2020On the Outsized Importance of Learning Rates in Local Update Methodscited by 34
- 2020Privacy Amplification by Decentralizationcited by 28
- 2018On the Optimal Recovery Threshold of Coded Matrix Multiplicationcited by 27
- 2017Straggler Mitigation in Distributed Optimization Through Data Encodingcited by 26
- 2019Robust and Communication-Efficient Collaborative Learningcited by 14
- 2020Private Weighted Random Walk Stochastic Gradient Descentcited by 14
- 2021Basil: A Fast and Byzantine-Resilient Approach for Decentralized Trainingcited by 13
- 2017Block-Diagonal and LT Codes for Distributed Computing With Straggling Serverscited by 5
- 2021Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligentcited by 5
- 2021Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workerscited by 3
- 2019Timely-Throughput Optimal Coded Computing over Cloud Networkscited by 2
- 2018Decentralized Differentially Private Without-Replacement Stochastic Gradient Descentcited by 1
- 2019Differentially Private Consensus-Based Distributed Optimizationcited by 1
- 2021DSAG: A mixed synchronous-asynchronous iterative method for straggler-resilient learningcited by 1
Built on by
Citing papers per year: 2023: 1; 2024: 2.
3 papers.