mlsanityData to 7 Oct 2026

Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

arXiv 2007.13237 ↗First posted 26 Jul 2020cs.IRcited by 48

Not on the map. Why?

Abstract (arXiv)

Effective methodologies for evaluating recommender systems are critical, so that such systems can be compared in a sound manner. A commonly overlooked aspect of recommender system evaluation is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems. In particular, we perform experiments comparing three common splitting strategies, examining their impact over seven state-of-the-art recommendation models for two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of state-of-the-art systems, making much of the currently published literature non-comparable, even when the same dataset and metrics are used.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 10; 2024: 13; 2025: 11; 2026: 6.

48 papers.

Sort:

By area:

  1. 2022Rank List Sensitivity of Recommender Systems to Interaction Perturbationscited by 20
  2. 2022Unlearning Protected User Attributes in Recommendations with Adversarial Trainingcited by 19
  3. 2024Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendationscited by 15
  4. 2024Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogscited by 15
  5. 2024From Variability to Stability: Advancing RecSys Benchmarking Practicescited by 12
  6. 2025Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenderscited by 12
  7. 2024RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenderscited by 9
  8. 2022On Sampling Collaborative Filtering Datasetscited by 8
  9. 2022Implicit Session Contexts for Next-Item Recommendationscited by 5
  10. 2022Tensor-based Sequential Learning via Hankel Matrix Representation for Next Item Recommendationscited by 5
  11. 2024RAGent: Retrieval-based Access Control Policy Generationcited by 5
  12. 2022Data Augmentation Strategies for Improving Sequential Recommender Systemscited by 4
36 more
  1. 2023When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?cited by 4
  2. 2024Green Recommender Systems: Optimizing Dataset Size for Energy-Efficient Algorithm Performancecited by 4
  3. 2024Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Modelscited by 4
  4. 2023Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?cited by 3
  5. 2025Stairway to Fairness: Connecting Group and Individual Fairnesscited by 3
  6. 2023Survey of Federated Learning Models for Spatial-Temporal Mobility Applicationscited by 2
  7. 2023Impression-Informed Multi-Behavior Recommender System: A Hierarchical Graph Attention Approachcited by 2
  8. 2025Describe What You See with Multimodal Large Language Models to Enhance Video Recommendationscited by 2
  9. 2025Ultra Fast Warm Start Solution for Graph Recommendationscited by 2
  10. 2025ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalizationcited by 2
  11. 2026Can Generative Recommendation Reach Cold Items? A Temporal Perspective on Semantic-ID Generationcited by 2
  12. 2023Improving Sequential Recommendation Models with an Enhanced Loss Functioncited by 1
  13. 2023KNNs of Semantic Encodings for Rating Predictioncited by 1
  14. 2023Sustainable transparency in Recommender Systems: Bayesian Ranking of Images for Explainabilitycited by 1
  15. 2023Capturing Popularity Trends: A Simplistic Non-Personalized Approach for Enhanced Item Recommendationcited by 1
  16. 2024ITEM: Improving Training and Evaluation of Message-Passing based GNNs for top-k recommendationcited by 1
  17. 2024Fair Augmentation for Graph Collaborative Filteringcited by 1
  18. 2025We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Latercited by 1
  19. 2025ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Testscited by 1
  20. 2026Measuring Individual User Fairness with User Similarity and Effectiveness Disparitycited by 1
  21. 2021Next Period Recommendation Reality Checkcited by 0
  22. 2022CVTT: Cross-Validation Through Timecited by 0
  23. 2023Job recommendations: benchmarking of collaborative filtering methods for classifiedscited by 0
  24. 2023STUDY: Socially Aware Temporally Causal Decoder Recommender Systemscited by 0
  25. 2024Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression modelscited by 0
  26. 2024From Data to Decisions: The Transformational Power of Machine Learning in Business Recommendationscited by 0
  27. 2024Quantifying User Coherence: A Unified Framework for Analyzing Recommender Systems Across Domainscited by 0
  28. 2024Double Machine Learning for Adaptive Causal Representation in High-Dimensional Datacited by 0
  29. 2025Predictively Combatting Toxicity in Health-related Online Discussions through Machine Learningcited by 0
  30. 2025Far From Sight, Far From Mind: Inverse Distance Weighting for Graph Federated Recommendationcited by 0
  31. 2025Point of Interest Recommendation: Pitfalls and Viable Solutionscited by 0
  32. 2025Leveraging Geometric Insights in Hyperbolic Triplet Loss for Improved Recommendationscited by 0
  33. 2026A Bandit-Based Approach to Educational Recommender Systems: Contextual Thompson Sampling for Learner Skill Gain Optimizationcited by 0
  34. 2026SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splitscited by 0
  35. 2026HORIZON: A Benchmark for In-the-wild User Behaviour Modelingcited by 0
  36. 2026An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendationcited by 0