# Generalized Delayed Feedback Model with Post-Click Information in Recommender Systems

arXiv: https://arxiv.org/abs/2206.00407 · HTML: https://mlsanity.com/paper/2206.00407/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Machine Learning for Survival Analysis: A Survey](https://mlsanity.com/paper/1708.04649/) (2017)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [Joint Optimization of Tree-based Index and Deep Model for Recommender Systems](https://mlsanity.com/paper/1902.07565/) (2019)
- [Measuring the Business Value of Recommender Systems](https://mlsanity.com/paper/1908.08328/) (2019)
- [Adapting to Delays and Data in Adversarial Multi-Armed Bandits](https://mlsanity.com/paper/2010.06022/) (2020)
- [Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction](https://mlsanity.com/paper/1907.06558/) (2019)
- [Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising](https://mlsanity.com/paper/1907.10235/) (2019)
- [Hierarchically Modeling Micro and Macro Behaviors via Multi-Task Learning for Conversion Rate Prediction](https://mlsanity.com/paper/2104.09713/) (2021)
- [Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling](https://mlsanity.com/paper/2104.14121/) (2021)
- [A Nonparametric Delayed Feedback Model for Conversion Rate Prediction](https://mlsanity.com/paper/1802.00255/) (2018)
- [Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling](https://mlsanity.com/paper/2012.03245/) (2020)
- [A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback](https://mlsanity.com/paper/2002.02068/) (2020)
- [Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction](https://mlsanity.com/paper/2202.06472/) (2022)
- [Modeling Perception Errors towards Robust Decision Making in Autonomous Vehicles](https://mlsanity.com/paper/2001.11695/) (2020)
- [Delayed Feedback Modeling for the Entire Space Conversion Rate Prediction](https://mlsanity.com/paper/2011.11826/) (2020)
- [Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback](https://mlsanity.com/paper/2108.06167/) (2021)

## Built on by (most cited first, 9 of 9)

- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [Capturing Conversion Rate Fluctuation during Sales Promotions: A Novel Historical Data Reuse Approach](https://mlsanity.com/paper/2305.12837/) (2023)
- [COURIER: Contrastive User Intention Reconstruction for Large-Scale Visual Recommendation](https://mlsanity.com/paper/2306.05001/) (2023)
- [Learning from Delayed Feedback in Games via Extra Prediction](https://mlsanity.com/paper/2509.22426/) (2025)
- [Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions](https://mlsanity.com/paper/2512.01171/) (2025)
- [Follow the TRACE: Exploiting Post-Click Trajectories for Online Delayed Conversion Rate Prediction](https://mlsanity.com/paper/2604.23197/) (2026)
- [TWICE: Two-Clock, Two-Window Learning for Long-Horizon Conversion Prediction in Online Advertising](https://mlsanity.com/paper/2607.25404/) (2026)
- [DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation](https://mlsanity.com/paper/2609.02170/) (2026)
- [Delay Complexity Dimension: Learning with Delayed and Censored Labels](https://mlsanity.com/paper/2610.18935/) (2026)

## Abstract (arXiv)

Predicting conversion rate (e.g., the probability that a user will purchase an item) is a fundamental problem in machine learning based recommender systems. However, accurate conversion labels are revealed after a long delay, which harms the timeliness of recommender systems. Previous literature concentrates on utilizing early conversions to mitigate such a delayed feedback problem. In this paper, we show that post-click user behaviors are also informative to conversion rate prediction and can be used to improve timeliness. We propose a generalized delayed feedback model (GDFM) that unifies both post-click behaviors and early conversions as stochastic post-click information, which could be utilized to train GDFM in a streaming manner efficiently. Based on GDFM, we further establish a novel perspective that the performance gap introduced by delayed feedback can be attributed to a temporal gap and a sampling gap. Inspired by our analysis, we propose to measure the quality of post-click information with a combination of temporal distance and sample complexity. The training objective is re-weighted accordingly to highlight informative and timely signals. We validate our analysis on public datasets, and experimental performance confirms the effectiveness of our method.