# Democratizing online controlled experiments at Booking.com

arXiv: https://arxiv.org/abs/1710.08217 · HTML: https://mlsanity.com/paper/1710.08217/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2017.

## Built on by (most cited first, 6 of 6)

- [E-Commerce Promotions Personalization via Online Multiple-Choice Knapsack with Uplift Modeling](https://mlsanity.com/paper/2108.13298/) (2021)
- [Uplift Modeling: from Causal Inference to Personalization](https://mlsanity.com/paper/2308.09066/) (2023)
- [Incremental Profit per Conversion: a Response Transformation for Uplift Modeling in E-Commerce Promotions](https://mlsanity.com/paper/2306.13759/) (2023)
- [Designing an Interpretable Interface for Contextual Bandits](https://mlsanity.com/paper/2409.15143/) (2024)
- [Maturity Framework for Enhancing Machine Learning Quality](https://mlsanity.com/paper/2502.15758/) (2025)
- [Incentive Alignment in Online Experimentation](https://mlsanity.com/paper/2610.05922/) (2026)

## Abstract (arXiv)

There is an extensive literature about online controlled experiments, both on the statistical methods available to analyze experiment results as well as on the infrastructure built by several large scale Internet companies but also on the organizational challenges of embracing online experiments to inform product development. At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind. In this paper we explain how building a central repository of successes and failures to allow for knowledge sharing, having a generic and extensible code library which enforces a loose coupling between experimentation and business logic, monitoring closely and transparently the quality and the reliability of the data gathering pipelines to build trust in the experimentation infrastructure, and putting in place safeguards to enable anyone to have end to end ownership of their experiments have allowed such a large organization as Booking.com to truly and successfully democratize experimentation.