# Gradient Masked Federated Optimization

arXiv: https://arxiv.org/abs/2104.10322 · HTML: https://mlsanity.com/paper/2104.10322/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 11 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 10 of 10)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [On the Convergence of Local Descent Methods in Federated Learning](https://mlsanity.com/paper/1910.14425/) (2019)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)
- [Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating](https://mlsanity.com/paper/1910.08234/) (2019)
- [Predicting with High Correlation Features](https://mlsanity.com/paper/1910.00164/) (2019)

## Built on by (most cited first, 2 of 2)

- [FL Games: A Federated Learning Framework for Distribution Shifts](https://mlsanity.com/paper/2211.00184/) (2022)
- [FL Games: A federated learning framework for distribution shifts](https://mlsanity.com/paper/2205.11101/) (2022)

## Abstract (arXiv)

Federated Averaging (FedAVG) has become the most popular federated learning algorithm due to its simplicity and low communication overhead. We use simple examples to show that FedAVG has the tendency to sew together the optima across the participating clients. These sewed optima exhibit poor generalization when used on a new client with new data distribution. Inspired by the invariance principles in (Arjovsky et al., 2019; Parascandolo et al., 2020), we focus on learning a model that is locally optimal across the different clients simultaneously. We propose a modification to FedAVG algorithm to include masked gradients (AND-mask from (Parascandolo et al., 2020)) across the clients and uses them to carry out an additional server model update. We show that this algorithm achieves better accuracy (out-of-distribution) than FedAVG, especially when the data is non-identically distributed across clients.