# Estimating Wage Disparities Using Foundation Models

arXiv: https://arxiv.org/abs/2409.09894 · HTML: https://mlsanity.com/paper/2409.09894/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 78 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Semiparametric doubly robust targeted double machine learning: a review](https://mlsanity.com/paper/2203.06469/) (2022)
- [Difficult Lessons on Social Prediction from Wisconsin Public Schools](https://mlsanity.com/paper/2304.06205/) (2023)
- [Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulations](https://mlsanity.com/paper/1909.02210/) (2019)
- [Using Sequences of Life-events to Predict Human Lives](https://mlsanity.com/paper/2306.03009/) (2023)
- [Long Story Short: Omitted Variable Bias in Causal Machine Learning](https://mlsanity.com/paper/2112.13398/) (2021)
- [Sharp instruments for classifying compliers and generalizing causal effects](https://mlsanity.com/paper/1801.03635/) (2018)
- [CAREER: A Foundation Model for Labor Sequence Data](https://mlsanity.com/paper/2202.08370/) (2022)
- [Ultra-high Dimensional Variable Selection for Doubly Robust Causal Inference](https://mlsanity.com/paper/2007.14190/) (2020)

## Built on by (most cited first, 4 of 4)

- [Program Evaluation with Remotely Sensed Outcomes](https://mlsanity.com/paper/2411.10959/) (2024)
- [Allocating Human Oversight in AI-Enabled Analytics](https://mlsanity.com/paper/2604.12497/) (2026)
- [Learning Treatment Representations for Downstream Instrumental Variable Regression](https://mlsanity.com/paper/2506.02200/) (2025)
- [Wage Sentiment Indices Derived from Survey Comments via Large Language Models](https://mlsanity.com/paper/2509.00290/) (2025)

## Abstract (arXiv)

The rise of foundation models marks a paradigm shift in machine learning: instead of training specialized models from scratch, foundation models are first trained on massive datasets before being adapted or fine-tuned to make predictions on smaller datasets. Initially developed for text, foundation models have also excelled at making predictions about social science data. However, while many estimation problems in the social sciences use prediction as an intermediate step, they ultimately require different criteria for success. In this paper, we develop methods for fine-tuning foundation models to perform these estimation problems. We first characterize an omitted variable bias that can arise when a foundation model is only fine-tuned to maximize predictive accuracy. We then provide a novel set of conditions for fine-tuning under which estimates derived from a foundation model are root-n-consistent. Based on this theory, we develop new fine-tuning algorithms that empirically mitigate this omitted variable bias. To demonstrate our ideas, we study gender wage decomposition. This is a statistical estimation problem from econometrics where the goal is to decompose the gender wage gap into components that can and cannot be explained by career histories of workers. Classical methods for decomposing the wage gap employ simple predictive models of wages which condition on coarse summaries of career history that may omit factors that are important for explaining the gap. Instead, we use a custom-built foundation model to decompose the gender wage gap, which captures a richer representation of career history. Using data from the Panel Study of Income Dynamics, we find that career history explains more of the gender wage gap than standard econometric models can measure, and we identify elements of career history that are omitted by standard models but are important for explaining the wage gap.