# Learning Classifiers under Delayed Feedback with a Time Window Assumption

arXiv: https://arxiv.org/abs/2009.13092 · HTML: https://mlsanity.com/paper/2009.13092/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2020.

## Built on by (most cited first, 7 of 7)

- [Handling many conversions per click in modeling delayed feedback](https://mlsanity.com/paper/2101.02284/) (2021)
- [Personalized Interpolation: Achieving Efficient Conversion Estimation with Flexible Optimization Windows](https://mlsanity.com/paper/2501.14103/) (2025)
- [Delayed Feedback Modeling with Influence Functions](https://mlsanity.com/paper/2502.01669/) (2025)
- [Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions](https://mlsanity.com/paper/2512.01171/) (2025)
- [Generalized Distribution-Free Semi-Supervised Learning with Risk Rewrite](https://mlsanity.com/paper/2607.11947/) (2026)
- [TWICE: Two-Clock, Two-Window Learning for Long-Horizon Conversion Prediction in Online Advertising](https://mlsanity.com/paper/2607.25404/) (2026)
- [DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation](https://mlsanity.com/paper/2609.02170/) (2026)

## Abstract (arXiv)

We consider training a binary classifier under delayed feedback (DF learning). For example, in the conversion prediction in online ads, we initially receive negative samples that clicked the ads but did not buy an item; subsequently, some samples among them buy an item then change to positive. In the setting of DF learning, we observe samples over time, then learn a classifier at some point. We initially receive negative samples; subsequently, some samples among them change to positive. This problem is conceivable in various real-world applications such as online advertisements, where the user action takes place long after the first click. Owing to the delayed feedback, naive classification of the positive and negative samples returns a biased classifier. One solution is to use samples that have been observed for more than a certain time window assuming these samples are correctly labeled. However, existing studies reported that simply using a subset of all samples based on the time window assumption does not perform well, and that using all samples along with the time window assumption improves empirical performance. We extend these existing studies and propose a method with the unbiased and convex empirical risk that is constructed from all samples under the time window assumption. To demonstrate the soundness of the proposed method, we provide experimental results on a synthetic and open dataset that is the real traffic log datasets in online advertising.