# Exact Subspace Diffusion for Decentralized Multitask Learning

arXiv: https://arxiv.org/abs/2304.07358 · HTML: https://mlsanity.com/paper/2304.07358/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [NEXT: In-Network Nonconvex Optimization](https://mlsanity.com/paper/1602.00591/) (2016)
- [Graph signal processing for machine learning: A review and new perspectives](https://mlsanity.com/paper/2007.16061/) (2020)
- [On the Influence of Bias-Correction on Distributed Stochastic Optimization](https://mlsanity.com/paper/1903.10956/) (2019)
- [Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learning](https://mlsanity.com/paper/2001.02112/) (2020)
- [Dif-MAML: Decentralized Multi-Agent Meta-Learning](https://mlsanity.com/paper/2010.02870/) (2020)
- [On the Learning Behavior of Adaptive Networks - Part I: Transient Analysis](https://mlsanity.com/paper/1312.7581/) (2013)
- [DSA: Decentralized Double Stochastic Averaging Gradient Algorithm](https://mlsanity.com/paper/1506.04216/) (2015)
- [Proximity Without Consensus in Online Multi-Agent Optimization](https://mlsanity.com/paper/1606.05578/) (2016)
- [Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysis](https://mlsanity.com/paper/1702.05142/) (2017)
- [Distributed Learning in Non-Convex Environments -- Part II: Polynomial Escape from Saddle-Points](https://mlsanity.com/paper/1907.01849/) (2019)
- [Learning over Multitask Graphs -- Part I: Stability Analysis](https://mlsanity.com/paper/1805.08535/) (2018)
- [Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex Optimization](https://mlsanity.com/paper/1908.07023/) (2019)
- [On the Learning Behavior of Adaptive Networks - Part II: Performance Analysis](https://mlsanity.com/paper/1312.7580/) (2013)
- [Adaptation and learning over networks under subspace constraints -- Part I: Stability Analysis](https://mlsanity.com/paper/1905.08750/) (2019)
- [Adaptation and learning over networks under subspace constraints -- Part II: Performance Analysis](https://mlsanity.com/paper/1906.12250/) (2019)
- [Robustness and Convergence Analysis of First-Order Distributed Optimization Algorithms over Subspace Constraints](https://mlsanity.com/paper/2210.16277/) (2022)

## Abstract (arXiv)

Classical paradigms for distributed learning, such as federated or decentralized gradient descent, employ consensus mechanisms to enforce homogeneity among agents. While these strategies have proven effective in i.i.d. scenarios, they can result in significant performance degradation when agents follow heterogeneous objectives or data. Distributed strategies for multitask learning, on the other hand, induce relationships between agents in a more nuanced manner, and encourage collaboration without enforcing consensus. We develop a generalization of the exact diffusion algorithm for subspace constrained multitask learning over networks, and derive an accurate expression for its mean-squared deviation when utilizing noisy gradient approximations. We verify numerically the accuracy of the predicted performance expressions, as well as the improved performance of the proposed approach over alternatives based on approximate projections.