# A Critical Study on Data Leakage in Recommender System Offline Evaluation

arXiv: https://arxiv.org/abs/2010.11060 · HTML: https://mlsanity.com/paper/2010.11060/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 30 of 39)

- [GraphPro: Graph Pre-training and Prompt Learning for Recommendation](https://mlsanity.com/paper/2311.16716/) (2023)
- [Agentic Feedback Loop Modeling Improves Recommendation and User Simulation](https://mlsanity.com/paper/2410.20027/) (2024)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders](https://mlsanity.com/paper/2408.02354/) (2024)
- [Preference Diffusion for Recommendation](https://mlsanity.com/paper/2410.13117/) (2024)
- [Diffusion Generative Recommendation with Continuous Tokens](https://mlsanity.com/paper/2504.12007/) (2025)
- [Causality-Enhanced Behavior Sequence Modeling in LLMs for Personalized Recommendation](https://mlsanity.com/paper/2410.22809/) (2024)
- [Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendations](https://mlsanity.com/paper/2212.05720/) (2022)
- [Integrating Item Relevance in Training Loss for Sequential Recommender Systems](https://mlsanity.com/paper/2305.10824/) (2023)
- [Addressing Missing Data Issue for Diffusion-based Recommendation](https://mlsanity.com/paper/2505.12283/) (2025)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendation](https://mlsanity.com/paper/2509.26063/) (2025)
- [Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation](https://mlsanity.com/paper/2209.08642/) (2022)
- [LLM-KT: A Versatile Framework for Knowledge Transfer from Large Language Models to Collaborative Filtering](https://mlsanity.com/paper/2411.00556/) (2024)
- [Recommendation Is a Dish Better Served Warm](https://mlsanity.com/paper/2508.07856/) (2025)
- [Can Generative Recommendation Reach Cold Items? A Temporal Perspective on Semantic-ID Generation](https://mlsanity.com/paper/2607.21101/) (2026)
- [Leveraging Social Influence based on Users Activity Centers for Point-of-Interest Recommendation](https://mlsanity.com/paper/2201.03450/) (2022)
- [Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization](https://mlsanity.com/paper/2507.19473/) (2025)
- [A Metric for MLLM Alignment in Large-scale Recommendation](https://mlsanity.com/paper/2508.04963/) (2025)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Merging](https://mlsanity.com/paper/2601.15930/) (2026)
- [CVTT: Cross-Validation Through Time](https://mlsanity.com/paper/2205.05393/) (2022)
- [Advancements in Recommender Systems: A Comprehensive Analysis Based on Data, Algorithms, and Evaluation](https://mlsanity.com/paper/2407.18937/) (2024)
- [Quantifying User Coherence: A Unified Framework for Analyzing Recommender Systems Across Domains](https://mlsanity.com/paper/2410.02453/) (2024)
- [Improving Image Data Leakage Detection in Automotive Software](https://mlsanity.com/paper/2410.23312/) (2024)

## Abstract (arXiv)

Recommender models are hard to evaluate, particularly under offline setting. In this paper, we provide a comprehensive and critical analysis of the data leakage issue in recommender system offline evaluation. Data leakage is caused by not observing global timeline in evaluating recommenders, e.g., train/test data split does not follow global timeline. As a result, a model learns from the user-item interactions that are not expected to be available at prediction time. We first show the temporal dynamics of user-item interactions along global timeline, then explain why data leakage exists for collaborative filtering models. Through carefully designed experiments, we show that all models indeed recommend future items that are not available at the time point of a test instance, as the result of data leakage. The experiments are conducted with four widely used baseline models - BPR, NeuMF, SASRec, and LightGCN, on four popular offline datasets - MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic, adopting leave-last-one-out data split. We further show that data leakage does impact models' recommendation accuracy. Their relative performance orders thus become unpredictable with different amount of leaked future data in training. To evaluate recommendation systems in a realistic manner in offline setting, we propose a timeline scheme, which calls for a revisit of the recommendation model design.