# Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks

arXiv: https://arxiv.org/abs/1912.12716 · HTML: https://mlsanity.com/paper/1912.12716/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2019.

## Built on by (most cited first, 30 of 79)

- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges](https://mlsanity.com/paper/2201.08135/) (2022)
- [An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning](https://mlsanity.com/paper/2302.07173/) (2023)
- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [CrowdGuard: Federated Backdoor Detection in Federated Learning](https://mlsanity.com/paper/2210.07714/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Robust Training in High Dimensions via Block Coordinate Geometric Median Descent](https://mlsanity.com/paper/2106.08882/) (2021)
- [Defending against the Label-flipping Attack in Federated Learning](https://mlsanity.com/paper/2207.01982/) (2022)
- [A Huber Loss Minimization Approach to Byzantine Robust Federated Learning](https://mlsanity.com/paper/2308.12581/) (2023)
- [Beyond Traditional Threats: A Persistent Backdoor Attack on Federated Learning](https://mlsanity.com/paper/2404.17617/) (2024)
- [FL-Defender: Combating Targeted Attacks in Federated Learning](https://mlsanity.com/paper/2207.00872/) (2022)
- [Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challenges](https://mlsanity.com/paper/2407.06754/) (2024)
- [Synergizing Foundation Models and Federated Learning: A Survey](https://mlsanity.com/paper/2406.12844/) (2024)
- [BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning](https://mlsanity.com/paper/2104.06685/) (2021)
- [Bridging Differential Privacy and Byzantine-Robustness via Model Aggregation](https://mlsanity.com/paper/2205.00107/) (2022)
- [Secure Distributed Training at Scale](https://mlsanity.com/paper/2106.11257/) (2021)
- [Federated Learning Attacks and Defenses: A Survey](https://mlsanity.com/paper/2211.14952/) (2022)
- [Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Rates](https://mlsanity.com/paper/2310.09804/) (2023)
- [Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysis](https://mlsanity.com/paper/2110.07567/) (2021)
- [Federated Robustness Propagation: Sharing Robustness in Heterogeneous Federated Learning](https://mlsanity.com/paper/2106.10196/) (2021)
- [Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity](https://mlsanity.com/paper/2305.18285/) (2023)
- [Byzantine-Robust Federated Linear Bandits](https://mlsanity.com/paper/2204.01155/) (2022)
- [Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environment](https://mlsanity.com/paper/2307.07980/) (2023)
- [FLTracer: Accurate Poisoning Attack Provenance in Federated Learning](https://mlsanity.com/paper/2310.13424/) (2023)
- [Byzantine Fault-Tolerance in Federated Local SGD under 2f-Redundancy](https://mlsanity.com/paper/2108.11769/) (2021)
- [Long-Short History of Gradients is All You Need: Detecting Malicious and Unreliable Clients in Federated Learning](https://mlsanity.com/paper/2208.10273/) (2022)
- [BOBA: Byzantine-Robust Federated Learning with Label Skewness](https://mlsanity.com/paper/2208.12932/) (2022)
- [A Secure Federated Learning Framework for Residential Short Term Load Forecasting](https://mlsanity.com/paper/2209.14547/) (2022)
- [Robust Distributed Optimization With Randomly Corrupted Gradients](https://mlsanity.com/paper/2106.14956/) (2021)
- [BEV-SGD: Best Effort Voting SGD for Analog Aggregation Based Federated Learning against Byzantine Attackers](https://mlsanity.com/paper/2110.09660/) (2021)

## Abstract (arXiv)

This paper deals with distributed finite-sum optimization for learning over networks in the presence of malicious Byzantine attacks. To cope with such attacks, most resilient approaches so far combine stochastic gradient descent (SGD) with different robust aggregation rules. However, the sizeable SGD-induced stochastic gradient noise makes it challenging to distinguish malicious messages sent by the Byzantine attackers from noisy stochastic gradients sent by the 'honest' workers. This motivates us to reduce the variance of stochastic gradients as a means of robustifying SGD in the presence of Byzantine attacks. To this end, the present work puts forth a Byzantine attack resilient distributed (Byrd-) SAGA approach for learning tasks involving finite-sum optimization over networks. Rather than the mean employed by distributed SAGA, the novel Byrd- SAGA relies on the geometric median to aggregate the corrected stochastic gradients sent by the workers. When less than half of the workers are Byzantine attackers, the robustness of geometric median to outliers enables Byrd-SAGA to attain provably linear convergence to a neighborhood of the optimal solution, with the asymptotic learning error determined by the number of Byzantine workers. Numerical tests corroborate the robustness to various Byzantine attacks, as well as the merits of Byrd- SAGA over Byzantine attack resilient distributed SGD.