mlsanityData to 7 Oct 2026

Secure Distributed Training at Scale

arXiv 2106.11257 ↗First posted 21 Jun 2021cs.LG cs.DC math.OCcited by 11

Not on the map. Why?

Abstract (arXiv)

Many areas of deep learning benefit from using increasingly larger neural networks trained on public data, as is the case for pre-trained models for NLP and computer vision. Training such models requires a lot of computational resources (e.g., HPC clusters) that are not available to small research groups and independent researchers. One way to address it is for several smaller groups to pool their computational resources together and train a model that benefits all participants. Unfortunately, in this case, any participant can jeopardize the entire training run by sending incorrect updates, deliberately or by mistake. Training in presence of such peers requires specialized distributed training algorithms with Byzantine tolerance. These algorithms often sacrifice efficiency by introducing redundant communication or passing all updates through a trusted server, making it infeasible to apply them to large-scale deep learning, where models can have billions of parameters. In this work, we propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 52 of its 111 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

52 of 111 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
38 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 826
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  4. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  6. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  7. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  8. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  9. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  10. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 324
  11. 2019Robust Aggregation for Federated Learningcited by 314
  12. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  13. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  14. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  15. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  16. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  17. 2018RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasetscited by 148
  18. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  19. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  20. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 136
  21. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  22. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  23. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  24. 2018Byzantine Stochastic Gradient Descentcited by 81
  25. 2019Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attackscited by 79
  26. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 77
  27. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 74
  28. 2019Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Recordscited by 70
  29. 2020Linearly Converging Error Compensated SGDcited by 61
  30. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 56
  31. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  32. 2019BRIDGE: Byzantine-resilient Decentralized Gradient Descentcited by 45
  33. 2019DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregationcited by 44
  34. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  35. 2017ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learningcited by 36
  36. 2020Byzantine-Robust Decentralized Stochastic Optimization over Static and Time-Varying Networkscited by 10
  37. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 8
  38. 2020GossipSub: Attack-Resilient Message Propagation in the Filecoin and ETH2.0 Networkscited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 1; 2025: 2; 2026: 1.

11 papers.

  1. 2022Byzantine-Robust Decentralized Learning via ClippedGossipcited by 29
  2. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  3. 2023Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneitycited by 8
  4. 2022Training Transformers Togethercited by 5
  5. 2023Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differencescited by 5
  6. 2023Byzantine-Robust Loopless Stochastic Variance-Reduced Gradientcited by 3
  7. 2024Byzantine-Robust Gossip: Insights from a Dual Approachcited by 1
  8. 2022Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Topcited by 0
  9. 2025Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Releasecited by 0
  10. 2025Robust and Efficient Collaborative Learningcited by 0
  11. 2026Byzantine-Robust Optimization under (L0, L1)-Smoothnesscited by 0