mlsanityData to 14 Oct 2026

Data splitting improves statistical performance in overparametrized regimes

arXiv 2110.10956 ↗First posted 21 Oct 2021stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

While large training datasets generally offer improvement in model performance, the training process becomes computationally expensive and time consuming. Distributed learning is a common strategy to reduce the overall training time by exploiting multiple computing devices. Recently, it has been observed in the single machine setting that overparametrization is essential for benign overfitting in ridgeless regression in Hilbert spaces. We show that in this regime, data splitting has a regularizing effect, hence improving statistical performance and computational complexity at the same time. We further provide a unified framework that allows to analyze both the finite and infinite dimensional setting. We numerically demonstrate the effect of different model parameters.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

22 of 51 references.

References · 22

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2021Deep learning: a statistical viewpointcited by 240
  4. 2019Two models of double descent for weak featurescited by 195
  5. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  6. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  7. 2019Harmless interpolation of noisy data in regressioncited by 121
  8. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  9. 2013Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Ratescited by 65
  10. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  11. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  12. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
10 more
  1. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  2. 2016Distributed learning with regularized least squarescited by 42
  3. 2019Communication-Efficient Accurate Statistical Estimationcited by 32
  4. 2018Distributed linear regression by averagingcited by 21
  5. 2019Beating SGD Saturation with Tail-Averaging and Minibatchingcited by 21
  6. 2014On the Optimality of Averaging in Distributed Statistical Learningcited by 15
  7. 2019WONDER: Weighted one-shot distributed ridge regression in high dimensionscited by 14
  8. 2020On the Universality of the Double Descent Peak in Ridgeless Regressioncited by 11
  9. 2016Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and its Asymptotic Optimalitycited by 9
  10. 2020On the robustness of the minimum ℓ2 interpolatorcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 2.

6 papers.

  1. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  2. 2022Mitigating multiple descents: A model-agnostic framework for risk monotonizationcited by 10
  3. 2023Corrected generalized cross-validation for finite ensembles of penalized estimatorscited by 5
  4. 2022Learning Variational Models with Unrolling and Bilevel Optimizationcited by 3
  5. 2022Local SGD in Overparameterized Linear Regressioncited by 0
  6. 2023Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regressioncited by 0