mlsanityData to 14 Oct 2026

A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization

arXiv 2111.02355 ↗First posted 3 Nov 2021cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Covariate-shift generalization, a typical case in out-of-distribution (OOD) generalization, requires a good performance on the unknown test distribution, which varies from the accessible training distribution in the form of covariate shift. Recently, independence-driven importance weighting algorithms in stable learning literature have shown empirical effectiveness to deal with covariate-shift generalization on several learning models, including regression algorithms and deep neural networks, while their theoretical analyses are missing. In this paper, we theoretically prove the effectiveness of such algorithms by explaining them as feature selection processes. We first specify a set of variables, named minimal stable variable set, that is the minimal and optimal set of variables to deal with covariate-shift generalization for common loss functions, such as the mean squared loss and binary cross-entropy loss. Afterward, we prove that under ideal conditions, independence-driven importance weighting algorithms could identify the variables in this set. Analysis of asymptotic properties is also provided. These theories are further validated in several synthetic experiments.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 28 of its 104 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

28 of 104 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  3. 2021Domain Generalization: A Surveycited by 764
  4. 2018DAGs with NO TEARS: Continuous Optimization for Structure Learningcited by 752
  5. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  6. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  7. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  8. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  9. 2016Learning Representations for Counterfactual Inferencecited by 329
  10. 2019Learning Sparse Nonparametric DAGscited by 207
  11. 2013Causal Inference in the Presence of Latent Variables and Selection Biascited by 166
  12. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
16 more
  1. 2020A Unified View of Label Shift Estimationcited by 119
  2. 2020Rethinking Importance Weighting for Deep Learning under Distribution Shiftcited by 93
  3. 2021Heterogeneous Risk Minimizationcited by 90
  4. 2017Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discoverycited by 76
  5. 2019Stable Learning via Sample Reweightingcited by 70
  6. 2020Stable Prediction with Model Misspecification and Agnostic Distribution Shiftcited by 62
  7. 2020Distributionally Robust Losses for Latent Covariate Mixturescited by 58
  8. 2018Stable Prediction across Unknown Environmentscited by 55
  9. 2021Towards Unsupervised Domain Generalizationcited by 40
  10. 2017Relief-Based Feature Selection: Introduction and Reviewcited by 38
  11. 2019Stabilizing Variable Selection and Regressioncited by 31
  12. 2017Causally Regularized Learning with Agnostic Data Selection Biascited by 29
  13. 2021Kernelized Heterogeneous Risk Minimizationcited by 21
  14. 2018Feature Selection using Stochastic Gatescited by 12
  15. 2015Markov Boundary Discovery with Ridge Regularized Linear Modelscited by 4
  16. 2020Balance-Subsampled Stable Predictioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 2; 2025: 4; 2026: 4.

19 papers.

Sort:

By area:

  1. 2022Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shiftcited by 38
  2. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Modelscited by 30
  3. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  4. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 21
  5. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  6. 2023TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Samplingcited by 16
  7. 2022Stable Learning via Sparse Variable Independencecited by 13
  8. 2023Data Optimization in Deep Learning: A Surveycited by 6
  9. 2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 4
  10. 2024Stable Heterogeneous Treatment Effect Estimation across Out-of-Distribution Populationscited by 1
  11. 2025Class-Conditional Distribution Balancing for Group Robust Classificationcited by 1
  12. 2025Heterogeneous Data Game: Characterizing the Model Competition Across Multiple Data Sourcescited by 1
7 more
  1. 2025Decorrelated feature importance from local sample weightingcited by 1
  2. 2026Stable Time Series Prediction of Enterprise Carbon Emissions Based on Causal Inferencecited by 1
  3. 2023Prediction with Incomplete Data under Agnostic Mask Distribution Shiftcited by 0
  4. 2025Rule Learning for Knowledge Graph Reasoning under Agnostic Distribution Shiftcited by 0
  5. 2026StableRCA: Robust Graph-Agnostic Mechanism-Level Root Cause Analysiscited by 0
  6. 2026Decoupled Causal Discoverycited by 0
  7. 2026Latent-Space Importance Weighting under Covariate Shiftcited by 0