# Incentive Alignment in Online Experimentation

arXiv: https://arxiv.org/abs/2610.05922 · HTML: https://mlsanity.com/paper/2610.05922/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 10 of 10)

- [Optimal Inference After Model Selection](https://mlsanity.com/paper/1410.2597/) (2014)
- [Designing and Deploying Online Field Experiments](https://mlsanity.com/paper/1409.3174/) (2014)
- [Democratizing online controlled experiments at Booking.com](https://mlsanity.com/paper/1710.08217/) (2017)
- [Test & Roll: Profit-Maximizing A/B Tests](https://mlsanity.com/paper/1811.00457/) (2018)
- [Sharp Results for Hypothesis Testing with Risk-Sensitive Agents](https://mlsanity.com/paper/2412.16452/) (2024)
- [Instance-Adaptive Hypothesis Tests with Heterogeneous Agents](https://mlsanity.com/paper/2510.21178/) (2025)
- [The Significance Filter, the Winner's Curse and the Need to Shrink](https://mlsanity.com/paper/2009.09440/) (2020)
- [Optimizing Returns from Experimentation Programs](https://mlsanity.com/paper/2412.05508/) (2024)
- [Optimal Testing in the Experiment-rich Regime](https://mlsanity.com/paper/1805.11754/) (2018)
- [Breaking the Winner's Curse with Bayesian Hybrid Shrinkage](https://mlsanity.com/paper/2603.12867/) (2026)

## Abstract (arXiv)

Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who develop new features also dictate which hypotheses to test, and they are typically rewarded based on empirical average treatment effects that are prone to upward bias. Left unchecked, this principal-agent conflict can severely erode platform value, a structural failure that conventional centralized levers, such as significance thresholds and traffic budgets, cannot resolve. By reframing experimentation as an incentive design problem, we demonstrate that two practical mechanisms, sample splitting and shrinkage, can effectively bridge this gap. Sample splitting aligns incentives perfectly at a bounded traffic cost, while shrinkage consumes no additional traffic and guarantees that interventions with negative expected effects are strictly unprofitable to field.