# A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization

arXiv: https://arxiv.org/abs/2111.02355 · HTML: https://mlsanity.com/paper/2111.02355/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 104 references that are in the map.
- First posted on arXiv 3 Nov 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Learning Sparse Nonparametric DAGs](https://mlsanity.com/paper/1909.13189/) (2019)
- [Causal Inference in the Presence of Latent Variables and Selection Bias](https://mlsanity.com/paper/1302.4983/) (2013)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [Rethinking Importance Weighting for Deep Learning under Distribution Shift](https://mlsanity.com/paper/2006.04662/) (2020)
- [Heterogeneous Risk Minimization](https://mlsanity.com/paper/2105.03818/) (2021)
- [Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discovery](https://mlsanity.com/paper/1702.03877/) (2017)
- [Stable Learning via Sample Reweighting](https://mlsanity.com/paper/1911.12580/) (2019)
- [Stable Prediction with Model Misspecification and Agnostic Distribution Shift](https://mlsanity.com/paper/2001.11713/) (2020)
- [Distributionally Robust Losses for Latent Covariate Mixtures](https://mlsanity.com/paper/2007.13982/) (2020)
- [Stable Prediction across Unknown Environments](https://mlsanity.com/paper/1806.06270/) (2018)
- [Towards Unsupervised Domain Generalization](https://mlsanity.com/paper/2107.06219/) (2021)
- [Relief-Based Feature Selection: Introduction and Review](https://mlsanity.com/paper/1711.08421/) (2017)
- [Stabilizing Variable Selection and Regression](https://mlsanity.com/paper/1911.01850/) (2019)
- [Causally Regularized Learning with Agnostic Data Selection Bias](https://mlsanity.com/paper/1708.06656/) (2017)
- [Kernelized Heterogeneous Risk Minimization](https://mlsanity.com/paper/2110.12425/) (2021)
- [Feature Selection using Stochastic Gates](https://mlsanity.com/paper/1810.04247/) (2018)
- [Markov Boundary Discovery with Ridge Regularized Linear Models](https://mlsanity.com/paper/1509.03935/) (2015)
- [Balance-Subsampled Stable Prediction](https://mlsanity.com/paper/2006.04381/) (2020)

## Built on by (most cited first, 19 of 19)

- [Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift](https://mlsanity.com/paper/2211.02843/) (2022)
- [On the Out-Of-Distribution Generalization of Multimodal Large Language Models](https://mlsanity.com/paper/2402.06599/) (2024)
- [Flatness-Aware Minimization for Domain Generalization](https://mlsanity.com/paper/2307.11108/) (2023)
- [A Survey on the Robustness of Computer Vision Models against Common Corruptions](https://mlsanity.com/paper/2305.06024/) (2023)
- [Rethinking the Evaluation Protocol of Domain Generalization](https://mlsanity.com/paper/2305.15253/) (2023)
- [TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Sampling](https://mlsanity.com/paper/2310.01819/) (2023)
- [Stable Learning via Sparse Variable Independence](https://mlsanity.com/paper/2212.00992/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [Learning Generalizable Agents via Saliency-Guided Features Decorrelation](https://mlsanity.com/paper/2310.05086/) (2023)
- [Stable Heterogeneous Treatment Effect Estimation across Out-of-Distribution Populations](https://mlsanity.com/paper/2407.03082/) (2024)
- [Class-Conditional Distribution Balancing for Group Robust Classification](https://mlsanity.com/paper/2504.17314/) (2025)
- [Heterogeneous Data Game: Characterizing the Model Competition Across Multiple Data Sources](https://mlsanity.com/paper/2505.07688/) (2025)
- [Decorrelated feature importance from local sample weighting](https://mlsanity.com/paper/2508.06337/) (2025)
- [Stable Time Series Prediction of Enterprise Carbon Emissions Based on Causal Inference](https://mlsanity.com/paper/2602.00775/) (2026)
- [Prediction with Incomplete Data under Agnostic Mask Distribution Shift](https://mlsanity.com/paper/2305.11197/) (2023)
- [Rule Learning for Knowledge Graph Reasoning under Agnostic Distribution Shift](https://mlsanity.com/paper/2507.05110/) (2025)
- [StableRCA: Robust Graph-Agnostic Mechanism-Level Root Cause Analysis](https://mlsanity.com/paper/2606.05636/) (2026)
- [Decoupled Causal Discovery](https://mlsanity.com/paper/2609.23535/) (2026)
- [Latent-Space Importance Weighting under Covariate Shift](https://mlsanity.com/paper/2610.15571/) (2026)

## Abstract (arXiv)

Covariate-shift generalization, a typical case in out-of-distribution (OOD) generalization, requires a good performance on the unknown test distribution, which varies from the accessible training distribution in the form of covariate shift. Recently, independence-driven importance weighting algorithms in stable learning literature have shown empirical effectiveness to deal with covariate-shift generalization on several learning models, including regression algorithms and deep neural networks, while their theoretical analyses are missing. In this paper, we theoretically prove the effectiveness of such algorithms by explaining them as feature selection processes. We first specify a set of variables, named minimal stable variable set, that is the minimal and optimal set of variables to deal with covariate-shift generalization for common loss functions, such as the mean squared loss and binary cross-entropy loss. Afterward, we prove that under ideal conditions, independence-driven importance weighting algorithms could identify the variables in this set. Analysis of asymptotic properties is also provided. These theories are further validated in several synthetic experiments.