# WONDER: Weighted one-shot distributed ridge regression in high dimensions

arXiv: https://arxiv.org/abs/1903.09321 · HTML: https://mlsanity.com/paper/1903.09321/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Demystifying Disagreement-on-the-Line in High Dimensions](https://mlsanity.com/paper/2301.13371/) (2023)
- [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023)
- [Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learning](https://mlsanity.com/paper/2406.11151/) (2024)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression](https://mlsanity.com/paper/2103.05621/) (2021)
- [Data splitting improves statistical performance in overparametrized regimes](https://mlsanity.com/paper/2110.10956/) (2021)
- [On Optimal Early Stopping: Over-informative versus Under-informative Parametrization](https://mlsanity.com/paper/2202.09885/) (2022)
- [Transfer Learning with Random Coefficient Ridge Regression](https://mlsanity.com/paper/2306.15915/) (2023)
- [Transfer learning via Regularized Linear Discriminant Analysis](https://mlsanity.com/paper/2501.02411/) (2025)
- [Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients](https://mlsanity.com/paper/2110.01595/) (2021)
- [Local SGD in Overparameterized Linear Regression](https://mlsanity.com/paper/2210.11562/) (2022)
- [Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regression](https://mlsanity.com/paper/2306.08432/) (2023)
- [Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing](https://mlsanity.com/paper/2602.16531/) (2026)

## Abstract (arXiv)

In many areas, practitioners need to analyze large datasets that challenge conventional single-machine computing. To scale up data analysis, distributed and parallel computing approaches are increasingly needed. Here we study a fundamental and highly important problem in this area: How to do ridge regression in a distributed computing environment? Ridge regression is an extremely popular method for supervised learning, and has several optimality properties, thus it is important to study. We study one-shot methods that construct weighted combinations of ridge regression estimators computed on each machine. By analyzing the mean squared error in a high dimensional random-effects model where each predictor has a small effect, we discover several new phenomena. 1. Infinite-worker limit: The distributed estimator works well for very large numbers of machines, a phenomenon we call "infinite-worker limit". 2. Optimal weights: The optimal weights for combining local estimators sum to more than unity, due to the downward bias of ridge. Thus, all averaging methods are suboptimal. We also propose a new Weighted ONe-shot DistributEd Ridge regression (WONDER) algorithm. We test WONDER in simulation studies and using the Million Song Dataset as an example. There it can save at least 100x in computation time, while nearly preserving test accuracy.