# Distributed Momentum for Byzantine-resilient Learning

arXiv: https://arxiv.org/abs/2003.00010 · HTML: https://mlsanity.com/paper/2003.00010/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 10 of 10)

- [Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments](https://mlsanity.com/paper/2107.07334/) (2021)
- [Byzantine-robust Federated Learning through Spatial-temporal Analysis of Local Model Updates](https://mlsanity.com/paper/2107.01477/) (2021)
- [Byzantine-Robust Distributed Online Learning: Taming Adversarial Participants in An Adversarial Environment](https://mlsanity.com/paper/2307.07980/) (2023)
- [Dynamic Byzantine-Robust Learning: Adapting to Switching Byzantine Workers](https://mlsanity.com/paper/2402.02951/) (2024)
- [LEGATO: A LayerwisE Gradient AggregaTiOn Algorithm for Mitigating Byzantine Attacks in Federated Learning](https://mlsanity.com/paper/2107.12490/) (2021)
- [FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning](https://mlsanity.com/paper/2303.05206/) (2023)
- [Byzantine-Robust Decentralized Stochastic Optimization with Stochastic Gradient Noise-Independent Learning Error](https://mlsanity.com/paper/2308.05292/) (2023)
- [Inverting Gradient Attacks Makes Powerful Data Poisoning](https://mlsanity.com/paper/2410.21453/) (2024)
- [Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning](https://mlsanity.com/paper/2106.06891/) (2021)
- [Byzantine-Robust Federated Learning via Credibility Assessment on Non-IID Data](https://mlsanity.com/paper/2109.02396/) (2021)

## Abstract (arXiv)

Momentum is a variant of gradient descent that has been proposed for its benefits on convergence. In a distributed setting, momentum can be implemented either at the server or the worker side. When the aggregation rule used by the server is linear, commutativity with addition makes both deployments equivalent. Robustness and privacy are however among motivations to abandon linear aggregation rules. In this work, we demonstrate the benefits on robustness of using momentum at the worker side. We first prove that computing momentum at the workers reduces the variance-norm ratio of the gradient estimation at the server, strengthening Byzantine resilient aggregation rules. We then provide an extensive experimental demonstration of the robustness effect of worker-side momentum on distributed SGD.