# Bayes beats Cross Validation: Efficient and Accurate Ridge Regression via Expectation Maximization

arXiv: https://arxiv.org/abs/2310.18860 · HTML: https://mlsanity.com/paper/2310.18860/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 7 of 7)

- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [HYDRA: Competing convolutional kernels for fast and accurate time series classification](https://mlsanity.com/paper/2203.13652/) (2022)
- [Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization](https://mlsanity.com/paper/1805.10939/) (2018)
- [Can we globally optimize cross-validation loss? Quasiconvexity in ridge regression](https://mlsanity.com/paper/2107.09194/) (2021)
- [Sparse Horseshoe Estimation via Expectation-Maximisation](https://mlsanity.com/paper/2211.03248/) (2022)

## Built on by (most cited first, 1 of 1)

- [Efficient Parameter Estimation for Bayesian Network Classifiers using Hierarchical Linear Smoothing](https://mlsanity.com/paper/2505.23320/) (2025)

## Abstract (arXiv)

We present a novel method for tuning the regularization hyper-parameter, λ, of a ridge regression that is faster to compute than leave-one-out cross-validation (LOOCV) while yielding estimates of the regression parameters of equal, or particularly in the setting of sparse covariates, superior quality to those obtained by minimising the LOOCV risk. The LOOCV risk can suffer from multiple and bad local minima for finite n and thus requires the specification of a set of candidate λ, which can fail to provide good solutions. In contrast, we show that the proposed method is guaranteed to find a unique optimal solution for large enough n, under relatively mild conditions, without requiring the specification of any difficult to determine hyper-parameters. This is based on a Bayesian formulation of ridge regression that we prove to have a unimodal posterior for large enough n, allowing for both the optimal λ and the regression coefficients to be jointly learned within an iterative expectation maximization (EM) procedure. Importantly, we show that by utilizing an appropriate preprocessing step, a single iteration of the main EM loop can be implemented in O(min(n, p)) operations, for input data with n rows and p columns. In contrast, evaluating a single value of λ using fast LOOCV costs O(n min(n, p)) operations when using the same preprocessing. This advantage amounts to an asymptotic improvement of a factor of l for l candidate values for λ (in the regime q, p ∈ O(√n) where q is the number of regression targets).