# Distributed learning with regularized least squares

arXiv: https://arxiv.org/abs/1608.03339 · HTML: https://mlsanity.com/paper/1608.03339/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2016.

## Built on by (most cited first, 30 of 42)

- [On the Optimality of Misspecified Spectral Algorithms](https://mlsanity.com/paper/2303.14942/) (2023)
- [Distribution Regression with Sliced Wasserstein Kernels](https://mlsanity.com/paper/2202.03926/) (2022)
- [Capacity dependent analysis for functional online learning algorithms](https://mlsanity.com/paper/2209.12198/) (2022)
- [Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling](https://mlsanity.com/paper/2111.14008/) (2021)
- [Statistical Optimality of Divide and Conquer Kernel-based Functional Linear Regression](https://mlsanity.com/paper/2211.10968/) (2022)
- [Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks](https://mlsanity.com/paper/2106.00881/) (2021)
- [ParK: Sound and Efficient Kernel Ridge Regression by Feature Space Partitions](https://mlsanity.com/paper/2106.12231/) (2021)
- [Data splitting improves statistical performance in overparametrized regimes](https://mlsanity.com/paper/2110.10956/) (2021)
- [Radial Basis Function Approximation with Distributively Stored Data on Spheres](https://mlsanity.com/paper/2112.02499/) (2021)
- [Random Smoothing Regularization in Kernel Gradient Descent Learning](https://mlsanity.com/paper/2305.03531/) (2023)
- [Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization](https://mlsanity.com/paper/2310.01139/) (2023)
- [Total Stability of SVMs and Localized SVMs](https://mlsanity.com/paper/2101.12678/) (2021)
- [Optimal Kernel Quantile Learning with Random Features](https://mlsanity.com/paper/2408.13591/) (2024)
- [Nystr"om Regularization for Time Series Forecasting](https://mlsanity.com/paper/2111.07109/) (2021)
- [Adaptive Parameter Selection for Kernel Ridge Regression](https://mlsanity.com/paper/2312.05885/) (2023)
- [Coefficient-based Regularized Distribution Regression](https://mlsanity.com/paper/2208.12427/) (2022)
- [Distributed Gradient Descent for Functional Learning](https://mlsanity.com/paper/2305.07408/) (2023)
- [Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Silos](https://mlsanity.com/paper/2309.04236/) (2023)
- [Distributed Uncertainty Quantification of Kernel Interpolation on Spheres](https://mlsanity.com/paper/2310.16384/) (2023)
- [Fixed-Gaussian Spectral Algorithms: Minimax Optimal Rates for Misspecified Learning and Transfer](https://mlsanity.com/paper/2501.10870/) (2025)
- [Sketching with Spherical Designs for Noisy Data Fitting on Spheres](https://mlsanity.com/paper/2303.04550/) (2023)
- [Distributed Semi-Supervised Sparse Statistical Inference](https://mlsanity.com/paper/2306.10395/) (2023)
- [On regularized polynomial functional regression](https://mlsanity.com/paper/2311.03036/) (2023)
- [Distributed Learning with Discretely Observed Functional Data](https://mlsanity.com/paper/2410.02376/) (2024)
- [Striking the Perfect Balance: Preserving Privacy While Boosting Utility in Collaborative Medical Prediction Platforms](https://mlsanity.com/paper/2507.11187/) (2025)
- [Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents](https://mlsanity.com/paper/2603.03401/) (2026)
- [Uncertainty quantification for distributed regression](https://mlsanity.com/paper/2105.11425/) (2021)
- [Oversampling Divide-and-conquer for Response-skewed Kernel Ridge Regression](https://mlsanity.com/paper/2107.05834/) (2021)
- [Local SGD in Overparameterized Linear Regression](https://mlsanity.com/paper/2210.11562/) (2022)
- [Kernel-Based Distributed Q-Learning: A Scalable Reinforcement Learning Approach for Dynamic Treatment Regimes](https://mlsanity.com/paper/2302.10434/) (2023)

## Abstract (arXiv)

We study distributed learning with the least squares regularization scheme in a reproducing kernel Hilbert space (RKHS). By a divide-and-conquer approach, the algorithm partitions a data set into disjoint data subsets, applies the least squares regularization scheme to each data subset to produce an output function, and then takes an average of the individual output functions as a final global estimator or predictor. We show with error bounds in expectation in both the L2-metric and RKHS-metric that the global output function of this distributed learning is a good approximation to the algorithm processing the whole data in one single machine. Our error bounds are sharp and stated in a general setting without any eigenfunction assumption. The analysis is achieved by a novel second order decomposition of operator differences in our integral operator approach. Even for the classical least squares regularization scheme in the RKHS associated with a general kernel, we give the best learning rate in the literature.