# Smooth Sequential Optimisation with Delayed Feedback

arXiv: https://arxiv.org/abs/2106.11294 · HTML: https://mlsanity.com/paper/2106.11294/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 25 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 4 of 4)

- [Linear Bandits with Stochastic Delayed Feedback](https://mlsanity.com/paper/1807.02089/) (2018)
- [Best arm identification in multi-armed bandits with delayed feedback](https://mlsanity.com/paper/1803.10937/) (2018)
- [Shrinkage Estimators in Online Experiments](https://mlsanity.com/paper/1904.12918/) (2019)
- [Stochastic Bandits with Delayed Composite Anonymous Feedback](https://mlsanity.com/paper/1910.01161/) (2019)

## Built on by (most cited first, 2 of 2)

- [Dynamic Memory for Interpretable Sequential Optimisation](https://mlsanity.com/paper/2206.13960/) (2022)
- [Rapid and Scalable Bayesian AB Testing](https://mlsanity.com/paper/2307.14628/) (2023)

## Abstract (arXiv)

Stochastic delays in feedback lead to unstable sequential learning using multi-armed bandits. Recently, empirical Bayesian shrinkage has been shown to improve reward estimation in bandit learning. Here, we propose a novel adaptation to shrinkage that estimates smoothed reward estimates from windowed cumulative inputs, to deal with incomplete knowledge from delayed feedback and non-stationary rewards. Using numerical simulations, we show that this adaptation retains the benefits of shrinkage, and improves the stability of reward estimation by more than 50%. Our proposal reduces variability in treatment allocations to the best arm by up to 3.8x, and improves statistical accuracy - with up to 8% improvement in true positive rates and 37% reduction in false positive rates. Together, these advantages enable control of the trade-off between speed and stability of adaptation, and facilitate human-in-the-loop sequential optimisation.