mlsanityData to 7 Oct 2026

Robust Distributed Optimization With Randomly Corrupted Gradients

arXiv 2106.14956 ↗First posted 28 Jun 2021math.OC cs.DC cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a first-order distributed optimization algorithm that is provably robust to Byzantine failures-arbitrary and potentially adversarial behavior, where all the participating agents are prone to failure. We model each agent's state over time as a two-state Markov chain that indicates Byzantine or trustworthy behaviors at different time instants. We set no restrictions on the maximum number of Byzantine agents at any given time. We design our method based on three layers of defense: 1) temporal robust aggregation, 2) spatial robust aggregation, and 3) gradient normalization. We study two settings for stochastic optimization, namely Sample Average Approximation and Stochastic Approximation. We provide convergence guarantees of our method for strongly convex and smooth non-convex cost functions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 28 Jun 2021.

Builds on

29 of 58 references.

References · 29

  1. 2019Advances and Open Problems in Federated Learningcited by 3,021
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  4. 2018How To Backdoor Federated Learningcited by 745
  5. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  6. 2017Large Batch Training of Convolutional Networkscited by 529
  7. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  8. 2017Certified Defenses for Data Poisoning Attackscited by 288
  9. 2019Can You Really Backdoor Federated Learning?cited by 250
  10. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 182
  11. 2020Momentum Improves Normalized SGDcited by 136
  12. 2019Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attackscited by 79
17 more
  1. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 75
  2. 2018DRACO: Byzantine-resilient Distributed Training via Redundant Gradientscited by 74
  3. 2017ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learningcited by 36
  4. 2018Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learningcited by 32
  5. 2016The Power of Normalization: Faster Evasion of Saddle Pointscited by 29
  6. 2020Statistically Preconditioned Accelerated Gradient Method for Distributed Optimizationcited by 29
  7. 2015On the Convergence Rate of Incremental Aggregated Gradient Algorithmscited by 21
  8. 2019On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyondcited by 18
  9. 2018The Internet of Things: Secure Distributed Inferencecited by 9
  10. 2016Distributed stochastic optimization via matrix exponential learningcited by 6
  11. 2016Distributed Learning for Stochastic Generalized Nash Equilibrium Problemscited by 6
  12. 2013Distributed Online Learning via Cooperative Contextual Banditscited by 5
  13. 2021Hyperfast Second-Order Local Solvers for Efficient Statistically Preconditioned Distributed Optimizationcited by 5
  14. 2020Resilient Primal-Dual Optimization Algorithms for Distributed Resource Allocationcited by 3
  15. 2019Gradient Methods for Problems with Inexact Model of the Objectivecited by 2
  16. 2013Empirical Centroid Fictitious Play: An Approach For Distributed Learning In Multi-Agent Gamescited by 1
  17. 2020On Robustness of the Normalized Subgradient Method with Randomly Corrupted Subgradientscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2.

6 papers.

  1. 2022AFLGuard: Byzantine-robust Asynchronous Federated Learningcited by 29
  2. 2024Federated Learning Resilient to Byzantine Attacks and Data Heterogeneitycited by 4
  3. 2023SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruningcited by 2
  4. 2024Efficient Federated Learning against Byzantine Attacks and Data Heterogeneity via Aggregating Normalized Gradientscited by 2
  5. 2022Communication-Efficient {Federated} Learning Using Censored Heavy Ball Descentcited by 1
  6. 2023Federated Learning Robust to Byzantine Attacks: Achieving Zero Optimality Gapcited by 0