# Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions

arXiv: https://arxiv.org/abs/1807.02694 · HTML: https://mlsanity.com/paper/1807.02694/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2018.

## Built on by (most cited first, 15 of 15)

- [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023)
- [Dissecting the Impact of Model Misspecification in Data-driven Optimization](https://mlsanity.com/paper/2503.00626/) (2025)
- [Iterative Approximate Cross-Validation](https://mlsanity.com/paper/2303.02732/) (2023)
- [Certified Data Removal Under High-dimensional Settings](https://mlsanity.com/paper/2505.07640/) (2025)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stability](https://mlsanity.com/paper/2305.19694/) (2023)
- [Optimizer's Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimization](https://mlsanity.com/paper/2306.10081/) (2023)
- [Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent](https://mlsanity.com/paper/2402.16793/) (2024)
- [Newfluence: Boosting Model interpretability and Understanding in High Dimensions](https://mlsanity.com/paper/2507.11895/) (2025)
- [Automatic Hyperparameter Tuning in Sparse Matrix Factorization](https://mlsanity.com/paper/2305.10114/) (2023)
- [Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference](https://mlsanity.com/paper/2501.02624/) (2025)
- [Benign overfitting beyond prediction: The ordinary least squares interpolator](https://mlsanity.com/paper/2309.15769/) (2023)
- [Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data](https://mlsanity.com/paper/2408.03138/) (2024)
- [Evaluation and Optimization of Leave-one-out Cross-validation for the Lasso](https://mlsanity.com/paper/2508.14368/) (2025)
- [Fermat Active Laplace Learning for Semi-Supervised Hyperspectral Image Classification](https://mlsanity.com/paper/2608.02483/) (2026)

## Abstract (arXiv)

Consider the following class of learning schemes: hatboldsymbolβ := argmin_boldsymbolβ;sumj=1n ℓ(boldsymbolxj^topboldsymbolβ; yj) + λ R(boldsymbolβ),qquadqquad (1)  where boldsymbolxi ∈ ℝp and yi ∈ ℝ denote the ith feature and response variable respectively. Let ℓ and R be the loss function and regularizer, boldsymbolβ denote the unknown weights, and λ be a regularization parameter. Finding the optimal choice of λ is a challenging problem in high-dimensional regimes where both n and p are large. We propose two frameworks to obtain a computationally efficient approximation ALO of the leave-one-out cross validation (LOOCV) risk for nonsmooth losses and regularizers. Our two frameworks are based on the primal and dual formulations of (1). We prove the equivalence of the two approaches under smoothness conditions. This equivalence enables us to justify the accuracy of both methods under such conditions. We use our approaches to obtain a risk estimate for several standard problems, including generalized LASSO, nuclear norm regularization, and support vector machines. We empirically demonstrate the effectiveness of our results for non-differentiable cases.