# Linear Regression with Distributed Learning: A Generalization Error Perspective

arXiv: https://arxiv.org/abs/2101.09001 · HTML: https://mlsanity.com/paper/2101.09001/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Federated Learning for Wireless Communications: Motivation, Opportunities and Challenges](https://mlsanity.com/paper/1908.06847/) (2019)
- [Harmless interpolation of noisy data in regression](https://mlsanity.com/paper/1903.09139/) (2019)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [CoCoA: A General Framework for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1611.02189/) (2016)
- [Communication-Efficient Distributed Dual Coordinate Ascent](https://mlsanity.com/paper/1409.1458/) (2014)
- [Distributed Optimization with Arbitrary Local Solvers](https://mlsanity.com/paper/1512.04039/) (2015)
- [Privacy-preserving Distributed Machine Learning via Local Randomization and ADMM Perturbation](https://mlsanity.com/paper/1908.01059/) (2019)
- [Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing](https://mlsanity.com/paper/2010.00914/) (2020)
- [Distributed Clustering and Learning Over Networks](https://mlsanity.com/paper/1409.6111/) (2014)
- [Distributed Constrained Online Learning](https://mlsanity.com/paper/1903.06310/) (2019)
- [Generalization Error for Linear Regression under Distributed Learning](https://mlsanity.com/paper/2004.14637/) (2020)

## Built on by (most cited first, 4 of 4)

- [Estimation under Model Misspecification with Fake Features](https://mlsanity.com/paper/2203.03398/) (2022)
- [Continual Learning with Distributed Optimization: Does CoCoA Forget?](https://mlsanity.com/paper/2211.16994/) (2022)
- [Federated Coordinate Descent for Privacy-Preserving Multiparty Linear Regression](https://mlsanity.com/paper/2209.07702/) (2022)
- [Distributed Continual Learning with CoCoA in High-dimensional Linear Regression](https://mlsanity.com/paper/2312.01795/) (2023)

## Abstract (arXiv)

Distributed learning provides an attractive framework for scaling the learning task by sharing the computational load over multiple nodes in a network. Here, we investigate the performance of distributed learning for large-scale linear regression where the model parameters, i.e., the unknowns, are distributed over the network. We adopt a statistical learning approach. In contrast to works that focus on the performance on the training data, we focus on the generalization error, i.e., the performance on unseen data. We provide high-probability bounds on the generalization error for both isotropic and correlated Gaussian data as well as sub-gaussian data. These results reveal the dependence of the generalization performance on the partitioning of the model over the network. In particular, our results show that the generalization error of the distributed solution can be substantially higher than that of the centralized solution even when the error on the training data is at the same level for both the centralized and distributed approaches. Our numerical results illustrate the performance with both real-world image data as well as synthetic data.