mlsanityData to 7 Oct 2026

An Information-theoretic Approach to Distribution Shifts

arXiv 2106.03783 ↗First posted 7 Jun 2021cs.LG cs.IT math.ITcited by 25

Not on the map. Why?

Abstract (arXiv)

Safely deploying machine learning models to the real world is often a challenging process. Models trained with data obtained from a specific geographic location tend to fail when queried with data obtained elsewhere, agents trained in a simulation can struggle to adapt when deployed in the real world or novel environments, and neural networks that are fit to a subset of the population might carry some selection bias into their decision process. In this work, we describe the problem of data shift from a novel information-theoretic perspective by (i) identifying and describing the different sources of error, (ii) comparing some of the most promising objectives explored in the recent domain generalization, and fair classification literature. From our theoretical analysis and empirical evaluation, we conclude that the model selection procedure needs to be guided by careful considerations regarding the observed data, the factors used for correction, and the structure of the data-generating process.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 33 of its 58 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

33 of 58 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  5. 2019Invariant Risk Minimizationcited by 1,832
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  8. 2020In Search of Lost Domain Generalizationcited by 1,042
  9. 2016Deep Variational Information Bottleneckcited by 915
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  11. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  12. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
21 more
  1. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  2. 2013Domain Generalization via Invariant Feature Representationcited by 622
  3. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  4. 2017Algorithmic decision making and the cost of fairnesscited by 402
  5. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  6. 2017Functional Map of the Worldcited by 338
  7. 2020The Risks of Invariant Risk Minimizationcited by 262
  8. 2020Domain Generalization using Causal Matchingcited by 224
  9. 2015The Variational Fair Autoencodercited by 223
  10. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  11. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  12. 2020Learning explanations that are hard to varycited by 132
  13. 2017Controllable Invariance through Adversarial Feature Learningcited by 127
  14. 2017Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributionscited by 125
  15. 2019DIVA: Domain Invariant Variational Autoencoderscited by 114
  16. 2018Invariant Representations without Adversarial Trainingcited by 111
  17. 2020Deep Visual Domain Adaptationcited by 91
  18. 2019Support and Invertibility in Domain-Invariant Representationscited by 87
  19. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  20. 2016A statistical framework for fair predictive algorithmscited by 37
  21. 2020Risk Variance Penalizationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 2; 2024: 5; 2025: 8; 2026: 3.

25 papers.

Sort:

By area:

  1. 2022Handling Distribution Shifts on Graphs: An Invariance Perspectivecited by 163
  2. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  3. 2023Environment-Aware Dynamic Graph Learning for Out-of-Distribution Generalizationcited by 26
  4. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  5. 2022Generalizing to Evolving Domains with Latent Structure-Aware Sequential Autoencodercited by 18
  6. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  7. 2022Finding Diverse and Predictable Subgraphs for Graph Domain Generalizationcited by 9
  8. 2022Measuring and signing fairness as performance under multiple stakeholder distributionscited by 6
  9. 2024Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learningcited by 6
  10. 2025Domain Adaptation and Entanglement: an Optimal Transport Perspectivecited by 6
  11. 2025When Shift Happens - Confounding Is to Blamecited by 5
  12. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
13 more
  1. 2022Testing robustness of predictions of trained classifiers against naturally occurring perturbationscited by 2
  2. 2024When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insightscited by 2
  3. 2024Invariant Graph Learning Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 2
  4. 2025Evolving Graph Learning for Out-of-Distribution Generalization in Non-stationary Environmentscited by 2
  5. 2025Adaptive Estimation and Learning under Temporal Distribution Shiftcited by 1
  6. 2025Ivan-ISTD: Rethinking Cross-domain Heteroscedastic Noise Perturbations in Infrared Small Target Detectioncited by 1
  7. 2025Understanding Generalization in Role-Playing Models via Information Theorycited by 1
  8. 2026SigmaQuant: Hardware-Aware Heterogeneous Quantization Method for Edge DNN Inferencecited by 1
  9. 2026OT on the Map: Quantifying Domain Shifts in Geographic Spacecited by 1
  10. 2021A benchmark with decomposed distribution shifts for 360 monocular depth estimationcited by 0
  11. 2024Monitoring Machine Learning Forecasts for Platform Data Streamscited by 0
  12. 2025Delta-Audit: Explaining What Changes When Models Changecited by 0
  13. 2026Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitionscited by 0