# Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching

arXiv: https://arxiv.org/abs/2602.08892 · HTML: https://mlsanity.com/paper/2602.08892/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1906.03735/) (2019)
- [Debiasing In-Sample Policy Performance for Small-Data, Large-Scale Optimization](https://mlsanity.com/paper/2107.12438/) (2021)
- [A Flexible Defense Against the Winner's Curse](https://mlsanity.com/paper/2411.18569/) (2024)
- [Beating the Winner's Curse via Inference-Aware Policy Optimization](https://mlsanity.com/paper/2510.18161/) (2025)
- [PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data](https://mlsanity.com/paper/2507.20068/) (2025)
- [Policy Learning with Confidence](https://mlsanity.com/paper/2502.10653/) (2025)

## Built on by (most cited first, 3 of 3)

- [Robustness of Refugee-Matching Gains to Off-Policy Evaluation Choices](https://mlsanity.com/paper/2605.06686/) (2026)
- [Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agents](https://mlsanity.com/paper/2608.10441/) (2026)
- [The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Broke](https://mlsanity.com/paper/2609.26160/) (2026)

## Abstract (arXiv)

A major challenge in data-driven decision-making is accurate policy evaluation-i.e., guaranteeing that a learned decision-making policy achieves the promised benefits. A popular strategy is model-based policy evaluation, which estimates a model from data to infer counterfactual outcomes. This strategy is known to produce unwarrantedly optimistic estimates of the true benefit due to the winner's curse. We searched the recent literature on data-driven decision-making, identifying a sample of 55 papers published in the Management Science in the past decade; all but two relied on this flawed methodology. Several common justifications are provided: (1) the estimated models are accurate, stable, and well-calibrated, (2) the historical data uses random treatment assignment, (3) the model family is well-specified, and (4) the evaluation methodology uses sample splitting. Unfortunately, we show that no combination of these justifications avoids the winner's curse. First, we provide a theoretical analysis demonstrating that the winner's curse can cause large, spurious reported benefits even when all these justifications hold. Second, we perform a simulation study based on the recent and consequential data-driven refugee matching problem. We construct a synthetic refugee matching environment (calibrated to closely match the real setting) but designed so that no assignment policy can improve expected employment compared to random assignment. Model-based methods report large, stable gains of around 60% even when the true effect is zero; these gains are on par with improvements of 22-75% reported in the literature. Our results provide strong evidence against model-based evaluation.