# Out-of-sample error estimate for robust M-estimators with convex penalty

arXiv: https://arxiv.org/abs/2008.11840 · HTML: https://mlsanity.com/paper/2008.11840/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality](https://mlsanity.com/paper/2309.16476/) (2023)
- [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023)
- [Derivatives and residual distribution of regularized M-estimators with application to adaptive tuning](https://mlsanity.com/paper/2107.05143/) (2021)
- [Asymptotic normality of robust M-estimators with convex penalty](https://mlsanity.com/paper/2107.03826/) (2021)
- [Observable adjustments in single-index models for regularized M-estimators](https://mlsanity.com/paper/2204.06990/) (2022)
- [A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond](https://mlsanity.com/paper/2205.10198/) (2022)
- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression](https://mlsanity.com/paper/2402.13622/) (2024)
- [Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent](https://mlsanity.com/paper/2402.16793/) (2024)
- [Noise Covariance Estimation in Multi-Task High-dimensional Linear Models](https://mlsanity.com/paper/2206.07256/) (2022)
- [Asymptotic Characterisation of Robust Empirical Risk Minimisation Performance in the Presence of Outliers](https://mlsanity.com/paper/2305.18974/) (2023)
- [Distributional bias compromises leave-one-out cross-validation](https://mlsanity.com/paper/2406.01652/) (2024)
- [Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression](https://mlsanity.com/paper/2410.02629/) (2024)

## Abstract (arXiv)

A generic out-of-sample error estimate is proposed for robust M-estimators regularized with a convex penalty in high-dimensional linear regression where (X,y) is observed and p,n are of the same order. If psi is the derivative of the robust data-fitting loss ρ, the estimate depends on the observed data only through the quantities hatpsi = psi(y-Xhatβ), X^top hatpsi and the derivatives (partial/partial y) hatpsi and (partial/partial y) Xhatβ for fixed X. The out-of-sample error estimate enjoys a relative error of order n-1/2 in a linear model with Gaussian covariates and independent noise, either non-asymptotically when p/n≤ γ or asymptotically in the high-dimensional asymptotic regime p/n→γ'∈(0,∞). General differentiable loss functions ρ are allowed provided that psi=ρ' is 1-Lipschitz. The validity of the out-of-sample error estimate holds either under a strong convexity assumption, or for the ℓ1-penalized Huber M-estimator if the number of corrupted observations and sparsity of the true β are bounded from above by s_*n for some small enough constant s_*∈(0,1) independent of n,p. For the square loss and in the absence of corruption in the response, the results additionally yield n-1/2-consistent estimates of the noise variance and of the generalization error. This generalizes, to arbitrary convex penalty, estimates that were previously known for the Lasso.