# Adaptive Experimentation with Delayed Binary Feedback

arXiv: https://arxiv.org/abs/2202.00846 · HTML: https://mlsanity.com/paper/2202.00846/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 7 of 7)

- [Online Learning under Delayed Feedback](https://mlsanity.com/paper/1306.0686/) (2013)
- [Linear Bandits with Stochastic Delayed Feedback](https://mlsanity.com/paper/1807.02089/) (2018)
- [Stochastic Bandit Models for Delayed Conversions](https://mlsanity.com/paper/1706.09186/) (2017)
- [Always Valid Inference: Bringing Sequential Analysis to A/B Testing](https://mlsanity.com/paper/1512.04922/) (2015)
- [Comparison Lift: Bandit-based Experimentation System for Online Advertising](https://mlsanity.com/paper/2009.07899/) (2020)
- [Online Evaluation of Audiences for Targeted Advertising via Bandit Experiments](https://mlsanity.com/paper/1907.02178/) (2019)
- [Multi-Armed Bandit Strategies for Non-Stationary Reward Distributions and Delayed Feedback Processes](https://mlsanity.com/paper/1902.08593/) (2019)

## Built on by (most cited first, 2 of 2)

- [Advertising Media and Target Audience Optimization via High-dimensional Bandits](https://mlsanity.com/paper/2209.08403/) (2022)
- [Continuous Attribution of Episodical Outcomes for More Efficient and Targeted Online Measurement](https://mlsanity.com/paper/2210.16373/) (2022)

## Abstract (arXiv)

Conducting experiments with objectives that take significant delays to materialize (e.g. conversions, add-to-cart events, etc.) is challenging. Although the classical "split sample testing" is still valid for the delayed feedback, the experiment will take longer to complete, which also means spending more resources on worse-performing strategies due to their fixed allocation schedules. Alternatively, adaptive approaches such as "multi-armed bandits" are able to effectively reduce the cost of experimentation. But these methods generally cannot handle delayed objectives directly out of the box. This paper presents an adaptive experimentation solution tailored for delayed binary feedback objectives by estimating the real underlying objectives before they materialize and dynamically allocating variants based on the estimates. Experiments show that the proposed method is more efficient for delayed feedback compared to various other approaches and is robust in different settings. In addition, we describe an experimentation product powered by this algorithm. This product is currently deployed in the online experimentation platform of JD.com, a large e-commerce company and a publisher of digital ads.