# Optimally tackling covariate shift in RKHS-based nonparametric regression arXiv: https://arxiv.org/abs/2205.02986 ยท HTML: https://mlsanity.com/paper/2205.02986/ ## Key facts - Cited by 54 papers in the map (data through 14 Oct 2026). - Builds on 7 of its 28 references that are in the map. - First posted on arXiv 6 May 2022. ## Builds on (most cited first, 7 of 7) - [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020) - [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021) - [Marginal Singularity, and the Benefits of Labels in Covariate-Shift](https://mlsanity.com/paper/1803.01833/) (2018) - [Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier](https://mlsanity.com/paper/1906.02903/) (2019) - [A new similarity measure for covariate shift with applications to nonparametric regression](https://mlsanity.com/paper/2202.02837/) (2022) - [Randomized sketches for kernels: Fast and optimal non-parametric regression](https://mlsanity.com/paper/1501.06195/) (2015) - [Minimax optimal approaches to the label shift problem in non-parametric settings](https://mlsanity.com/paper/2003.10443/) (2020) ## Built on by (most cited first, 30 of 54) - [Understanding Forgetting in Continual Learning with Linear Regression](https://mlsanity.com/paper/2405.17583/) (2024) - [The Exact Sample Complexity Gain from Invariances for Kernel Regression](https://mlsanity.com/paper/2303.14269/) (2023) - [The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift](https://mlsanity.com/paper/2208.01857/) (2022) - [Transfer Learning for Contextual Multi-armed Bandits](https://mlsanity.com/paper/2211.12612/) (2022) - [A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models](https://mlsanity.com/paper/2401.07187/) (2024) - [Pseudo-Labeling for Kernel Ridge Regression under Covariate Shift](https://mlsanity.com/paper/2302.10160/) (2023) - [TabPFN: One Model to Rule Them All?](https://mlsanity.com/paper/2505.20003/) (2025) - [When is Importance Weighting Correction Needed for Covariate Shift Adaptation?](https://mlsanity.com/paper/2303.04020/) (2023) - [Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift](https://mlsanity.com/paper/2310.08237/) (2023) - [Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks?](https://mlsanity.com/paper/2502.08991/) (2025) - [Utility Theory of Synthetic Data Generation](https://mlsanity.com/paper/2305.10015/) (2023) - [Nonlinear Meta-Learning Can Guarantee Faster Rates](https://mlsanity.com/paper/2307.10870/) (2023) - [Smoothness Adaptive Hypothesis Transfer Learning](https://mlsanity.com/paper/2402.14966/) (2024) - [Training-Conditional Coverage Bounds under Covariate Shift](https://mlsanity.com/paper/2405.16594/) (2024) - [Distance and Kernel-Based Measures for Global and Local Two-Sample Conditional Distribution Testing](https://mlsanity.com/paper/2210.08149/) (2022) - [Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency](https://mlsanity.com/paper/2301.06240/) (2023) - [Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation](https://mlsanity.com/paper/2310.16638/) (2023) - [Transfer Learning Beyond Bounded Density Ratios](https://mlsanity.com/paper/2403.11963/) (2024) - [Progression: an extrapolation principle for regression](https://mlsanity.com/paper/2410.23246/) (2024) - [Efficient and Multiply Robust Risk Estimation under General Forms of Dataset Shift](https://mlsanity.com/paper/2306.16406/) (2023) - [Tackling Combinatorial Distribution Shift: A Matrix Completion Perspective](https://mlsanity.com/paper/2307.06457/) (2023) - [Nearest Neighbor Sampling for Covariate Shift Adaptation](https://mlsanity.com/paper/2312.09969/) (2023) - [Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning](https://mlsanity.com/paper/2401.12216/) (2024) - [High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization](https://mlsanity.com/paper/2406.03171/) (2024) - [Computational Efficiency under Covariate Shift in Kernel Ridge Regression](https://mlsanity.com/paper/2505.14083/) (2025) - [Transfer Learning for Kernel-based Regression](https://mlsanity.com/paper/2310.13966/) (2023) - [On the design-dependent suboptimality of the Lasso](https://mlsanity.com/paper/2402.00382/) (2024) - [Distributionally robust risk evaluation with an isotonic constraint](https://mlsanity.com/paper/2407.06867/) (2024) - [Spectral Algorithms under Covariate Shift](https://mlsanity.com/paper/2504.12625/) (2025) - [Spectral Algorithms in Misspecified Regression: Convergence under Covariate Shift](https://mlsanity.com/paper/2509.05106/) (2025) ## Abstract (arXiv) We study the covariate shift problem in the context of nonparametric regression over a reproducing kernel Hilbert space (RKHS). We focus on two natural families of covariate shift problems defined using the likelihood ratios between the source and target distributions. When the likelihood ratios are uniformly bounded, we prove that the kernel ridge regression (KRR) estimator with a carefully chosen regularization parameter is minimax rate-optimal (up to a log factor) for a large family of RKHSs with regular kernel eigenvalues. Interestingly, KRR does not require full knowledge of likelihood ratios apart from an upper bound on them. In striking contrast to the standard statistical setting without covariate shift, we also demonstrate that a naive estimator, which minimizes the empirical risk over the function class, is strictly sub-optimal under covariate shift as compared to KRR. We then address the larger class of covariate shift problems where the likelihood ratio is possibly unbounded yet has a finite second moment. Here, we propose a reweighted KRR estimator that weights samples based on a careful truncation of the likelihood ratios. Again, we are able to show that this estimator is minimax rate-optimal, up to logarithmic factors.