# Beyond ATE: Multi-Criteria Design for A/B Testing

arXiv: https://arxiv.org/abs/2509.05864 · HTML: https://mlsanity.com/paper/2509.05864/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 83 references that are in the map.
- First posted on arXiv 6 Sep 2025.

## Builds on (most cited first, 30 of 44)

- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response](https://mlsanity.com/paper/1407.6981/) (2014)
- [Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/2002.04926/) (2020)
- [Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizability](https://mlsanity.com/paper/2003.12699/) (2020)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Differentially Private Contextual Linear Bandits](https://mlsanity.com/paper/1810.00068/) (2018)
- [Locally Differentially Private (Contextual) Bandits Learning](https://mlsanity.com/paper/2006.00701/) (2020)
- [The End of Optimism? An Asymptotic Analysis of Finite-Armed Linear Bandits](https://mlsanity.com/paper/1610.04491/) (2016)
- [Algorithms for Differentially Private Multi-Armed Bandits](https://mlsanity.com/paper/1511.08681/) (2015)
- [Adaptive Exploration in Linear Contextual Bandit](https://mlsanity.com/paper/1910.06996/) (2019)
- [Design and Analysis of Switchback Experiments](https://mlsanity.com/paper/2009.00148/) (2020)
- [An Optimal Private Stochastic-MAB Algorithm Based on an Optimal Private Stopping Rule](https://mlsanity.com/paper/1905.09383/) (2019)
- [Multi-Armed Bandits with Local Differential Privacy](https://mlsanity.com/paper/2007.03121/) (2020)
- [Why Adaptively Collected Data Have Negative Bias and How to Correct for It](https://mlsanity.com/paper/1708.01977/) (2017)
- [Power Constrained Bandits](https://mlsanity.com/paper/2004.06230/) (2020)
- [A Primer on Private Statistics](https://mlsanity.com/paper/2005.00010/) (2020)
- [Generalized Linear Bandits with Local Differential Privacy](https://mlsanity.com/paper/2106.03365/) (2021)
- [When Privacy Meets Partial Information: A Refined Analysis of Differentially Private Bandits](https://mlsanity.com/paper/2209.02570/) (2022)
- [Shuffle Private Linear Contextual Bandits](https://mlsanity.com/paper/2202.05567/) (2022)
- [Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimes](https://mlsanity.com/paper/1909.02553/) (2019)
- [Differentially Private Multi-Armed Bandits in the Shuffle Model](https://mlsanity.com/paper/2106.02900/) (2021)
- [Efficient Adaptive Experimental Design for Average Treatment Effect Estimation](https://mlsanity.com/paper/2002.05308/) (2020)
- [Privacy-Preserving Dynamic Personalized Pricing with Demand Learning](https://mlsanity.com/paper/2009.12920/) (2020)
- [Mitigating Bias in Adaptive Data Gathering via Differential Privacy](https://mlsanity.com/paper/1806.02329/) (2018)
- [Always Valid Inference: Bringing Sequential Analysis to A/B Testing](https://mlsanity.com/paper/1512.04922/) (2015)
- [Differentially Private Stochastic Linear Bandits: (Almost) for Free](https://mlsanity.com/paper/2207.03445/) (2022)
- [Causal Inference with Corrupted Data: Measurement Error, Missing Values, Discretization, and Differential Privacy](https://mlsanity.com/paper/2107.02780/) (2021)
- [A framework for Multi-A(rmed)/B(andit) testing with online FDR control](https://mlsanity.com/paper/1706.05378/) (2017)

## Abstract (arXiv)

In the era of large-scale AI deployment and high-stakes clinical trials, adaptive experimentation faces a ``trilemma'' of conflicting objectives: minimizing cumulative regret (welfare loss during the experiment), maximizing the estimation accuracy of heterogeneous treatment effects (CATE), and ensuring differential privacy (DP) for participants. Existing literature typically optimizes these metrics in isolation or under restrictive parametric assumptions. In this work, we study the multi-objective design of adaptive experiments in a general non-parametric setting. First, we rigorously characterize the instance-dependent Pareto frontier between cumulative regret and estimation error, revealing the fundamental statistical limits of dual-objective optimization. We propose ConSE, a sequential segmentation and elimination algorithm that adaptively discretizes the covariate space to achieve the Pareto-optimal frontier. Second, we introduce DP-ConSE, a privacy-preserving extension that satisfies Joint Differential Privacy. We demonstrate that privacy comes ``for free'' in our framework, incurring only asymptotically negligible costs to regret and estimation accuracy. Finally, we establish a robust link between experimental design and long-term utility: we prove that any policy derived from our Pareto-optimal algorithms minimizes post-experiment simple regret, regardless of the specific exploration-exploitation trade-off chosen during the trial. Our results provide a theoretical foundation for designing ethical, private, and efficient adaptive experiments in sensitive domains.