# Profit over Proxies: A Scalable Bayesian Decision Framework for Optimizing Multi-Variant Online Experiments

arXiv: https://arxiv.org/abs/2509.22677 · HTML: https://mlsanity.com/paper/2509.22677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 33 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 1 of 1)

- [Toward a principled Bayesian workflow in cognitive science](https://mlsanity.com/paper/1904.12765/) (2019)

## Abstract (arXiv)

Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy. However, their real-world application is frequently compromised by two critical shortcomings: the use of statistically flawed heuristics like "p-value peeking", which inflates false positive rates, and an over-reliance on proxy metrics like conversion rates, which can lead to decisions that inadvertently harm core business profitability. This paper addresses these challenges by introducing a comprehensive and scalable Bayesian decision framework designed for profit optimization in multi-variant (A/B/n) experiments. We propose a hierarchical Bayesian model that simultaneously estimates the probability of conversion (using a Beta-Bernoulli model) and the monetary value of that conversion (using a robust Bayesian model for the mean transaction value). Building on this, we employ a decision-theoretic stopping rule based on Expected Loss, enabling experiments to be concluded not only when a superior variant is identified but also when it becomes clear that no variant offers a practically significant improvement (stopping for futility). The framework successfully navigates "revenue traps" where a variant with a higher conversion rate would have resulted in a net financial loss, correctly terminates futile experiments early to conserve resources, and maintains strict statistical integrity throughout the monitoring process. Ultimately, this work provides a practical and principled methodology for organizations to move beyond simple A/B testing towards a mature, profit-driven experimentation culture, ensuring that statistical conclusions translate directly to strategic business value.