On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regression

We consider the linear model y = X β_star + ε with X∈ ℝn× p in the overparameterized regime p>n. We estimate β_star via generalized (weighted) ridge regression: hatβλ = left(XTX + λ Sigmawright)^dagger XTy, where Sigmaw is the weighting matrix. Under a random design setting with general data covariance Sigmax and anisotropic prior on the true coefficients mathbbEβ_starβ_starT = Sigmaβ, we provide an exact characterization of the prediction risk mathbbE(y-xThatβλ)2 in the proportional asymptotic limit p/nrightarrow γ ∈ (1,∞). Our general setup leads to a number of interesting findings. We outline precise conditions that decide the sign of the optimal setting λrm opt for the ridge parameter λ and confirm the implicit ℓ2 regularization effect of overparameterization, which theoretically justifies the surprising empirical observation that λrm opt can be negative in the overparameterized regime. We also characterize the double descent phenomenon for principal component regression (PCR) when both X and β_star are anisotropic. Finally, we determine the optimal weighting matrix Sigmaw for both the ridgeless (λ→ 0) and optimally regularized (λ = λrm opt) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.