# Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests

arXiv: https://arxiv.org/abs/2407.01036 · HTML: https://mlsanity.com/paper/2407.01036/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 6 of 6)

- [The Econometrics of Randomized Experiments](https://mlsanity.com/paper/1607.00698/) (2016)
- [Test & Roll: Profit-Maximizing A/B Tests](https://mlsanity.com/paper/1811.00457/) (2018)
- [Invidious Comparisons: Ranking and Selection as Compound Decisions](https://mlsanity.com/paper/2012.12550/) (2020)
- [Weighted False Discovery Rate Control in Large-Scale Multiple Testing](https://mlsanity.com/paper/1508.01605/) (2015)
- [Risk-aware product decisions in A/B tests with multiple metrics](https://mlsanity.com/paper/2402.11609/) (2024)
- [An Empirical Bayes Approach to Controlling the False Discovery Exceedance](https://mlsanity.com/paper/2111.03885/) (2021)

## Abstract (arXiv)

A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach.