# High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization

arXiv: https://arxiv.org/abs/2406.03171 · HTML: https://mlsanity.com/paper/2406.03171/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Just Interpolate: Kernel "Ridgeless" Regression Can Generalize](https://mlsanity.com/paper/1808.00387/) (2018)
- [Linearized two-layers neural networks in high dimension](https://mlsanity.com/paper/1904.12191/) (2019)
- [When Do Neural Networks Outperform Kernel Methods?](https://mlsanity.com/paper/2006.13409/) (2020)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration](https://mlsanity.com/paper/2101.10588/) (2021)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regression](https://mlsanity.com/paper/2006.05800/) (2020)
- [Rethinking Importance Weighting for Deep Learning under Distribution Shift](https://mlsanity.com/paper/2006.04662/) (2020)
- [Orthogonal Random Features](https://mlsanity.com/paper/1610.09072/) (2016)
- [Learning with invariances in random features and kernel models](https://mlsanity.com/paper/2102.13219/) (2021)
- [Marginal Singularity, and the Benefits of Labels in Covariate-Shift](https://mlsanity.com/paper/1803.01833/) (2018)
- [Kernel Alignment Risk Estimator: Risk Prediction from Training Data](https://mlsanity.com/paper/2006.09796/) (2020)
- [Optimally tackling covariate shift in RKHS-based nonparametric regression](https://mlsanity.com/paper/2205.02986/) (2022)
- [Kernel regression in high dimensions: Refined analysis beyond double descent](https://mlsanity.com/paper/2010.02681/) (2020)
- [Understanding Why Generalized Reweighting Does Not Improve Over ERM](https://mlsanity.com/paper/2201.12293/) (2022)
- [Towards a statistical theory of data selection under weak supervision](https://mlsanity.com/paper/2309.14563/) (2023)
- [Covariate Shift in High-Dimensional Random Feature Regression](https://mlsanity.com/paper/2111.08234/) (2021)
- [The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods](https://mlsanity.com/paper/2111.07167/) (2021)
- [Learning with convolution and pooling operations in kernel methods](https://mlsanity.com/paper/2111.08308/) (2021)
- [Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift](https://mlsanity.com/paper/2311.15961/) (2023)
- [Harmless interpolation in regression and classification with structured features](https://mlsanity.com/paper/2111.05198/) (2021)
- [When is Importance Weighting Correction Needed for Covariate Shift Adaptation?](https://mlsanity.com/paper/2303.04020/) (2023)
- [Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift](https://mlsanity.com/paper/2310.08237/) (2023)
- [Optimal Rate of Kernel Regression in Large Dimensions](https://mlsanity.com/paper/2309.04268/) (2023)
- [Strong inductive biases provably prevent harmless interpolation](https://mlsanity.com/paper/2301.07605/) (2023)

## Built on by (most cited first, 4 of 4)

- [Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning](https://mlsanity.com/paper/2408.16189/) (2024)
- [Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression](https://mlsanity.com/paper/2510.15337/) (2025)
- [Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning](https://mlsanity.com/paper/2507.04194/) (2025)
- [Optimal and Structure-Adaptive CATE Estimation with Kernel Ridge Regression](https://mlsanity.com/paper/2602.18958/) (2026)

## Abstract (arXiv)

This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposition, we theoretically demonstrate that the re-weighting strategy allows for decreasing the variance. For bias, we analyze the regularization of the arbitrary or well-chosen scale, showing that the bias can behave very differently under different regularization scales. In our analysis, the bias and variance can be characterized by the spectral decay of a data-dependent regularized kernel: the original kernel matrix associated with an additional re-weighting matrix, and thus the re-weighting strategy can be regarded as a data-dependent regularization for better understanding. Besides, our analysis provides asymptotic expansion of kernel functions/vectors under covariate shift, which has its own interest.