# Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback

arXiv: https://arxiv.org/abs/2108.06167 · HTML: https://mlsanity.com/paper/2108.06167/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Fast Matrix Factorization for Online Recommendation with Implicit Feedback](https://mlsanity.com/paper/1708.05024/) (2017)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction](https://mlsanity.com/paper/1907.06558/) (2019)
- [Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising](https://mlsanity.com/paper/1907.10235/) (2019)
- [A Nonparametric Delayed Feedback Model for Conversion Rate Prediction](https://mlsanity.com/paper/1802.00255/) (2018)
- [Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions](https://mlsanity.com/paper/1905.10713/) (2019)
- [A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback](https://mlsanity.com/paper/2002.02068/) (2020)
- [PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System](https://mlsanity.com/paper/1707.00802/) (2017)

## Built on by (most cited first, 5 of 5)

- [Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems](https://mlsanity.com/paper/2206.00407/) (2022)
- [Delayed Feedback Modeling with Influence Functions](https://mlsanity.com/paper/2502.01669/) (2025)
- [Freshness or Accuracy, Why Not Both? Addressing Delayed Feedback via Dynamic Graph Neural Networks](https://mlsanity.com/paper/2308.08071/) (2023)
- [Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions](https://mlsanity.com/paper/2512.01171/) (2025)
- [Follow the TRACE: Exploiting Post-Click Trajectories for Online Delayed Conversion Rate Prediction](https://mlsanity.com/paper/2604.23197/) (2026)

## Abstract (arXiv)

The delayed feedback problem is one of the imperative challenges in online advertising, which is caused by the highly diversified feedback delay of a conversion varying from a few minutes to several days. It is hard to design an appropriate online learning system under these non-identical delay for different types of ads and users. In this paper, we propose to tackle the delayed feedback problem in online advertising by "Following the Prophet" (FTP for short). The key insight is that, if the feedback came instantly for all the logged samples, we could get a model without delayed feedback, namely the "prophet". Although the prophet cannot be obtained during online learning, we show that we could predict the prophet's predictions by an aggregation policy on top of a set of multi-task predictions, where each task captures the feedback patterns of different periods. We propose the objective and optimization approach for the policy, and use the logged data to imitate the prophet. Extensive experiments on three real-world advertising datasets show that our method outperforms the previous state-of-the-art baselines.