# Omitted Variable Bias in Language Models Under Distribution Shift

arXiv: https://arxiv.org/abs/2602.16784 · HTML: https://mlsanity.com/paper/2602.16784/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [Change is Hard: A Closer Look at Subpopulation Shift](https://mlsanity.com/paper/2302.12254/) (2023)
- [MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations](https://mlsanity.com/paper/2502.06453/) (2025)
- [EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis](https://mlsanity.com/paper/2205.01996/) (2022)
- [Empath: Understanding Topic Signals in Large-Scale Text](https://mlsanity.com/paper/1602.06979/) (2016)
- [Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning](https://mlsanity.com/paper/2202.09667/) (2022)
- [Long Story Short: Omitted Variable Bias in Causal Machine Learning](https://mlsanity.com/paper/2112.13398/) (2021)
- [Reasoning Elicitation in Language Models via Counterfactual Feedback](https://mlsanity.com/paper/2410.03767/) (2024)
- [RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation](https://mlsanity.com/paper/2506.15455/) (2025)
- [Optimizing Language Models for Human Preferences is a Causal Inference Problem](https://mlsanity.com/paper/2402.14979/) (2024)
- [Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models](https://mlsanity.com/paper/2408.08210/) (2024)
- [Doubly Robust Counterfactual Classification](https://mlsanity.com/paper/2301.06199/) (2023)
- [SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations](https://mlsanity.com/paper/2305.14728/) (2023)
- [Isolated Causal Effects of Natural Language](https://mlsanity.com/paper/2410.14812/) (2024)

## Built on by (most cited first, 1 of 1)

- [Human-AI Collaboration for Estimating Scientific Replicability](https://mlsanity.com/paper/2605.27394/) (2026)

## Abstract (arXiv)

Despite their impressive performance on a wide variety of tasks, modern language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of language models under distribution shift. In empirical experiments, we show that using these bounds directly in language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.