mlsanityData to 14 Oct 2026

A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization

arXiv 2508.17412 ↗First posted 24 Aug 2025cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Conventional regularization is designed to control variance, but in small-data regression it can also aggravate underfitting when predictive signal is concentrated in weak directions of a restricted representation. We study a negative-capable ridge family that permits a feasible negative region whenever the estimator remains well posed, and show that negative regularization acts there as controlled anti-shrinkage by increasing effective complexity most strongly along weak eigendirections. Building on this mechanism, we formalize weak-spectrum underfitting, derive a sign-switch result under conservative baseline shrinkage, and study criterion-based automatic selection over the full negative-capable family. Synthetic and semi-synthetic experiments support the theory by verifying feasibility, spectral complexity increase, sign-switch behavior, and effective recovery of negative adjustments in the predicted regimes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 74 references that are in the map. First posted on arXiv 24 Aug 2025.

Builds on

27 of 74 references.

References · 27

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  4. 2020Benign overfitting in ridge regressioncited by 153
  5. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  6. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  8. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  9. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 89
  10. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 74
  11. 2023A Unified Theory of Diversity in Ensemble Learningcited by 71
  12. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 65
15 more
  1. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  2. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 53
  3. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
  4. 2017Early stopping for kernel boosting algorithms: A general analysis with localized complexitiescited by 27
  5. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  6. 2024Optimal Ridge Regularization for Out-of-Distribution Predictioncited by 19
  7. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  8. 2021Harmless interpolation in regression and classification with structured featurescited by 14
  9. 2020Neural Complexity Measurescited by 11
  10. 2022Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processescited by 11
  11. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 8
  12. 2024Risk and cross validation in ridge regression with correlated samplescited by 8
  13. 2023Iterative Approximate Cross-Validationcited by 7
  14. 2015Consistent Parameter Estimation for LASSO and Approximate Message Passingcited by 6
  15. 2021Asymptotics of Ridge Regression in Convolutional Modelscited by 3

Built on by

Not cited yet.