# Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning

arXiv: https://arxiv.org/abs/1806.05358 · HTML: https://mlsanity.com/paper/1806.05358/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 32)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Provably Secure Federated Learning against Malicious Clients](https://mlsanity.com/paper/2102.01854/) (2021)
- [Byzantine Machine Learning Made Easy by Resilient Averaging of Momentums](https://mlsanity.com/paper/2205.12173/) (2022)
- [Byzantine-Robust Decentralized Learning via ClippedGossip](https://mlsanity.com/paper/2202.01545/) (2022)
- [Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guarantees](https://mlsanity.com/paper/2205.11765/) (2022)
- [Streaming Algorithms for High-Dimensional Robust Statistics](https://mlsanity.com/paper/2204.12399/) (2022)
- [Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inference](https://mlsanity.com/paper/2103.02860/) (2021)
- [Robust Distributed Optimization With Randomly Corrupted Gradients](https://mlsanity.com/paper/2106.14956/) (2021)
- [Robust Federated Learning via Over-The-Air Computation](https://mlsanity.com/paper/2111.01221/) (2021)
- [Can Decentralized Learning be more robust than Federated Learning?](https://mlsanity.com/paper/2303.03829/) (2023)
- [SafeNet: The Unreasonable Effectiveness of Ensembles in Private Collaborative Learning](https://mlsanity.com/paper/2205.09986/) (2022)
- [MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks](https://mlsanity.com/paper/2207.07941/) (2022)
- [Secure and Verifiable Data Collaboration with Low-Cost Zero-Knowledge Proofs](https://mlsanity.com/paper/2311.15310/) (2023)
- [Over-The-Air Federated Learning under Byzantine Attacks](https://mlsanity.com/paper/2205.02949/) (2022)
- [FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning](https://mlsanity.com/paper/2303.05206/) (2023)
- [Escaping Saddle Points in Distributed Newton's Method with Communication Efficiency and Byzantine Resilience](https://mlsanity.com/paper/2103.09424/) (2021)
- [A simplified convergence theory for Byzantine resilient stochastic gradient descent](https://mlsanity.com/paper/2208.11879/) (2022)
- [BadSampler: Harnessing the Power of Catastrophic Forgetting to Poison Byzantine-robust Federated Learning](https://mlsanity.com/paper/2406.12222/) (2024)
- [Second-Order Convergence in Private Stochastic Non-Convex Optimization](https://mlsanity.com/paper/2505.15647/) (2025)
- [An Improved Algorithm for Clustered Federated Learning](https://mlsanity.com/paper/2210.11538/) (2022)
- [Robust Second-Order Nonconvex Optimization and Its Application to Low Rank Matrix Sensing](https://mlsanity.com/paper/2403.10547/) (2024)
- [Robust Gradient Descent for Phase Retrieval](https://mlsanity.com/paper/2410.10623/) (2024)
- [Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning](https://mlsanity.com/paper/2106.06891/) (2021)
- [Aspis: Robust Detection for Distributed Learning](https://mlsanity.com/paper/2108.02416/) (2021)
- [On the Conflict of Robustness and Learning in Collaborative Machine Learning](https://mlsanity.com/paper/2402.13700/) (2024)
- [Byzantine-tolerant distributed learning of finite mixture models](https://mlsanity.com/paper/2407.13980/) (2024)
- [Detection and Mitigation of Byzantine Attacks in Distributed Training](https://mlsanity.com/paper/2208.08085/) (2022)
- [On the Optimal Batch Size for Byzantine-Robust Distributed Learning](https://mlsanity.com/paper/2305.13856/) (2023)
- [Distributed quasi-Newton robust estimation under differential privacy](https://mlsanity.com/paper/2408.12353/) (2024)
- [Federated Independent Component Analysis via Spectral Alignment and Robust Aggregation](https://mlsanity.com/paper/2505.20532/) (2025)

## Abstract (arXiv)

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point that is far away from any true local minimum, even when robust gradient estimators are used. We develop ByzantinePGD, a robust first-order algorithm that can provably escape saddle points and fake local minima, and converge to an approximate true local minimizer with low iteration complexity. As a by-product, we give a simpler algorithm and analysis for escaping saddle points in the usual non-Byzantine setting. We further discuss three robust gradient estimators that can be used in ByzantinePGD, including median, trimmed mean, and iterative filtering. We characterize their performance in concrete statistical settings, and argue for their near-optimality in low and high dimensional regimes.