# Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

arXiv: https://arxiv.org/abs/2007.13237 · HTML: https://mlsanity.com/paper/2007.13237/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2020.

## Built on by (most cited first, 30 of 48)

- [Rank List Sensitivity of Recommender Systems to Interaction Perturbations](https://mlsanity.com/paper/2201.12686/) (2022)
- [Unlearning Protected User Attributes in Recommendations with Adversarial Training](https://mlsanity.com/paper/2206.04500/) (2022)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders](https://mlsanity.com/paper/2408.02354/) (2024)
- [On Sampling Collaborative Filtering Datasets](https://mlsanity.com/paper/2201.04768/) (2022)
- [Implicit Session Contexts for Next-Item Recommendations](https://mlsanity.com/paper/2208.09076/) (2022)
- [Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendations](https://mlsanity.com/paper/2212.05720/) (2022)
- [RAGent: Retrieval-based Access Control Policy Generation](https://mlsanity.com/paper/2409.07489/) (2024)
- [Data Augmentation Strategies for Improving Sequential Recommender Systems](https://mlsanity.com/paper/2203.14037/) (2022)
- [When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?](https://mlsanity.com/paper/2305.01801/) (2023)
- [Green Recommender Systems: Optimizing Dataset Size for Energy-Efficient Algorithm Performance](https://mlsanity.com/paper/2410.09359/) (2024)
- [Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models](https://mlsanity.com/paper/2410.17276/) (2024)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [Survey of Federated Learning Models for Spatial-Temporal Mobility Applications](https://mlsanity.com/paper/2305.05257/) (2023)
- [Impression-Informed Multi-Behavior Recommender System: A Hierarchical Graph Attention Approach](https://mlsanity.com/paper/2309.03169/) (2023)
- [Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations](https://mlsanity.com/paper/2508.09789/) (2025)
- [Ultra Fast Warm Start Solution for Graph Recommendations](https://mlsanity.com/paper/2509.01549/) (2025)
- [ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization](https://mlsanity.com/paper/2510.18433/) (2025)
- [Can Generative Recommendation Reach Cold Items? A Temporal Perspective on Semantic-ID Generation](https://mlsanity.com/paper/2607.21101/) (2026)
- [Improving Sequential Recommendation Models with an Enhanced Loss Function](https://mlsanity.com/paper/2301.00979/) (2023)
- [KNNs of Semantic Encodings for Rating Prediction](https://mlsanity.com/paper/2302.00412/) (2023)
- [Sustainable transparency in Recommender Systems: Bayesian Ranking of Images for Explainability](https://mlsanity.com/paper/2308.01196/) (2023)
- [Capturing Popularity Trends: A Simplistic Non-Personalized Approach for Enhanced Item Recommendation](https://mlsanity.com/paper/2308.08799/) (2023)
- [ITEM: Improving Training and Evaluation of Message-Passing based GNNs for top-k recommendation](https://mlsanity.com/paper/2407.07912/) (2024)
- [Fair Augmentation for Graph Collaborative Filtering](https://mlsanity.com/paper/2408.12208/) (2024)
- [We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Later](https://mlsanity.com/paper/2509.09414/) (2025)

## Abstract (arXiv)

Effective methodologies for evaluating recommender systems are critical, so that such systems can be compared in a sound manner. A commonly overlooked aspect of recommender system evaluation is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems. In particular, we perform experiments comparing three common splitting strategies, examining their impact over seven state-of-the-art recommendation models for two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of state-of-the-art systems, making much of the currently published literature non-comparable, even when the same dataset and metrics are used.