# Operationalizing Counterfactual Metrics: Incentives, Ranking, and Information Asymmetry

arXiv: https://arxiv.org/abs/2305.14595 · HTML: https://mlsanity.com/paper/2305.14595/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 78 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 17 of 17)

- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [Semiparametric doubly robust targeted double machine learning: a review](https://mlsanity.com/paper/2203.06469/) (2022)
- [Strategic Classification is Causal Modeling in Disguise](https://mlsanity.com/paper/1910.10362/) (2019)
- [How Do Classifiers Induce Agents To Invest Effort Strategically?](https://mlsanity.com/paper/1807.05307/) (2018)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Maximizing Welfare with Incentive-Aware Evaluation Mechanisms](https://mlsanity.com/paper/2011.01956/) (2020)
- [Bounds on the conditional and average treatment effect with unobserved confounding factors](https://mlsanity.com/paper/1808.09521/) (2018)
- [Simple versus Optimal Contracts](https://mlsanity.com/paper/1808.03713/) (2018)
- [On classification of strategic agents who can both game and improve](https://mlsanity.com/paper/2203.00124/) (2022)
- [The Complexity of Contracts](https://mlsanity.com/paper/2002.12034/) (2020)
- [Gaming Helps! Learning from Strategic Interactions in Natural Dynamics](https://mlsanity.com/paper/2002.07024/) (2020)
- [Principal-Agent Hypothesis Testing](https://mlsanity.com/paper/2205.06812/) (2022)
- [Counterfactual Prediction Under Outcome Measurement Error](https://mlsanity.com/paper/2302.11121/) (2023)
- [Approximate Optimality of Linear Contracts Under Uncertainty](https://mlsanity.com/paper/2211.06850/) (2022)
- [Policy Learning with Competing Agents](https://mlsanity.com/paper/2204.01884/) (2022)

## Built on by (most cited first, 2 of 2)

- [From Prototypes to Sparse ECG Explanations: SHAP-Driven Counterfactuals for Multivariate Time-Series Multi-class Classification](https://mlsanity.com/paper/2510.19514/) (2025)
- [Score Design for Multi-Criteria Incentivization](https://mlsanity.com/paper/2410.06290/) (2024)

## Abstract (arXiv)

From the social sciences to machine learning, it has been well documented that metrics to be optimized are not always aligned with social welfare. In healthcare, Dranove et al. (2003) showed that publishing surgery mortality metrics actually harmed the welfare of sicker patients by increasing provider selection behavior. We analyze the incentive misalignments that arise from such average treated outcome metrics, and show that the incentives driving treatment decisions would align with maximizing total patient welfare if the metrics (i) accounted for counterfactual untreated outcomes and (ii) considered total welfare instead of averaging over treated patients. Operationalizing this, we show how counterfactual metrics can be modified to behave reasonably in patient-facing ranking systems. Extending to realistic settings when providers observe more about patients than the regulatory agencies do, we bound the decay in performance by the degree of information asymmetry between principal and agent. In doing so, our model connects principal-agent information asymmetry with unobserved heterogeneity in causal inference.