# Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding

arXiv: https://arxiv.org/abs/2605.26067 · HTML: https://mlsanity.com/paper/2605.26067/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 17 of 17)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime](https://mlsanity.com/paper/2105.15004/) (2021)
- [KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation](https://mlsanity.com/paper/2205.09921/) (2022)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [Implicit Regularization of Random Feature Models](https://mlsanity.com/paper/2002.08404/) (2020)
- [Kernel Alignment Risk Estimator: Risk Prediction from Training Data](https://mlsanity.com/paper/2006.09796/) (2020)
- [Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance](https://mlsanity.com/paper/1902.03046/) (2019)
- [Eigenvalue distribution of nonlinear models of random matrices](https://mlsanity.com/paper/1904.03090/) (2019)
- [ResMem: Learn what you can and memorize the rest](https://mlsanity.com/paper/2302.01576/) (2023)
- [Operator World Models for Reinforcement Learning](https://mlsanity.com/paper/2406.19861/) (2024)
- [On the speed of uniform convergence in Mercer's theorem](https://mlsanity.com/paper/2205.00487/) (2022)
- [Multi-layer random features and the approximation power of neural networks](https://mlsanity.com/paper/2404.17461/) (2024)
- [On the Intrinsic Dimensions of Data in Kernel Learning](https://mlsanity.com/paper/2601.16139/) (2026)

## Abstract (arXiv)

Conditionally positive definite (CPD) kernels are defined with respect to a function class F. It is well known that such a kernel K is associated with its native space (defined analogously to an RKHS), which in turn gives rise to a learning method -- called conditional kernel ridge regression (conditional KRR) due to its analogy with KRR -- where the estimated regression function is penalized by the square of its native space norm. This method is of interest because it can be viewed as classical linear regression, with features specified by F, followed by the application of standard KRR to the residual (unexplained) component of the target variable. Methods of this type have recently attracted increasing attention. We study the statistical properties of this method by reducing its behavior to that of KRR with another fixed kernel, called the residual kernel. Our main theoretical result shows that such a reduction is indeed possible, at the cost of an additional term in the expected test risk, bounded by O(1/√N), where N is the sample size and the hidden constant depends on the class F and the input distribution. This reduction enables us to analyze conditional KRR in the case where K is positive definite and F is given by the first k principal eigenfunctions in the Mercer decomposition of K. We also consider the setting where F consists of k random features from a random feature representation of K. It turns out that these two settings are closely related. Both our theoretical analysis and experiments confirm that conditional KRR outperforms standard KRR in these cases whenever the F-component of the regression function is more pronounced than the residual part.