# Handling many conversions per click in modeling delayed feedback

arXiv: https://arxiv.org/abs/2101.02284 · HTML: https://mlsanity.com/paper/2101.02284/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [Stochastic Bandit Models for Delayed Conversions](https://mlsanity.com/paper/1706.09186/) (2017)
- [Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction](https://mlsanity.com/paper/1907.06558/) (2019)
- [A Nonparametric Delayed Feedback Model for Conversion Rate Prediction](https://mlsanity.com/paper/1802.00255/) (2018)
- [A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback](https://mlsanity.com/paper/2002.02068/) (2020)
- [Anti-Distillation: Improving reproducibility of deep networks](https://mlsanity.com/paper/2010.09923/) (2020)
- [Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles](https://mlsanity.com/paper/2001.11695/) (2020)
- [Learning Classifiers under Delayed Feedback with a Time Window Assumption](https://mlsanity.com/paper/2009.13092/) (2020)
- [Display advertising: Estimating conversion probability efficiently](https://mlsanity.com/paper/1710.08583/) (2017)
- [Beta Survival Models](https://mlsanity.com/paper/1905.03818/) (2019)

## Built on by (most cited first, 4 of 4)

- [Regression with Label Differential Privacy](https://mlsanity.com/paper/2212.06074/) (2022)
- [Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewards](https://mlsanity.com/paper/2206.01293/) (2022)
- [Multi-Head Online Learning for Delayed Feedback Modeling](https://mlsanity.com/paper/2205.12406/) (2022)
- [TWICE: Two-Clock, Two-Window Learning for Long-Horizon Conversion Prediction in Online Advertising](https://mlsanity.com/paper/2607.25404/) (2026)

## Abstract (arXiv)

Predicting the expected value or number of post-click conversions (purchases or other events) is a key task in performance-based digital advertising. In training a conversion optimizer model, one of the most crucial aspects is handling delayed feedback with respect to conversions, which can happen multiple times with varying delay. This task is difficult, as the delay distribution is different for each advertiser, is long-tailed, often does not follow any particular class of parametric distributions, and can change over time. We tackle these challenges using an unbiased estimation model based on three core ideas. The first idea is to split the label as a sum of labels with different delay buckets, each of which trains only on mature label, the second is to use thermometer encoding to increase accuracy and reduce inference cost, and the third is to use auxiliary information to increase the stability of the model and to handle drift in the distribution.