# SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits

arXiv: https://arxiv.org/abs/2602.19339 · HTML: https://mlsanity.com/paper/2602.19339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Improved Recurrent Neural Networks for Session-based Recommendations](https://mlsanity.com/paper/1606.08117/) (2016)
- [Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations](https://mlsanity.com/paper/2306.08121/) (2023)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [A Critical Study on Data Leakage in Recommender System Offline Evaluation](https://mlsanity.com/paper/2010.11060/) (2020)
- [Evaluation of Session-based Recommendation Algorithms](https://mlsanity.com/paper/1803.09587/) (2018)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [Take a Fresh Look at Recommender Systems from an Evaluation Standpoint](https://mlsanity.com/paper/2210.04149/) (2022)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs](https://mlsanity.com/paper/2409.18721/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Scaling Recommender Transformers to One Billion Parameters](https://mlsanity.com/paper/2507.15994/) (2025)
- [On Sampling Collaborative Filtering Datasets](https://mlsanity.com/paper/2201.04768/) (2022)
- [Autoregressive Generation Strategies for Top-K Sequential Recommendations](https://mlsanity.com/paper/2409.17730/) (2024)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [DataRec: A Python Library for Standardized and Reproducible Data Management in Recommender Systems](https://mlsanity.com/paper/2410.22972/) (2024)
- [Recommendation Is a Dish Better Served Warm](https://mlsanity.com/paper/2508.07856/) (2025)
- [Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization](https://mlsanity.com/paper/2507.19473/) (2025)

## Abstract (arXiv)

Offline evaluation of recommender systems is often affected by hidden, under-documented choices in data preparation. Seemingly minor decisions in filtering, handling repeats, cold-start treatment, and splitting strategy design can substantially reorder model rankings and undermine reproducibility and cross-paper comparability. In this paper, we introduce SplitLight, an open-source exploratory toolkit that enables researchers and practitioners designing preprocessing and splitting pipelines or reviewing external artifacts to make these decisions measurable, comparable, and reportable. Given an interaction log and derived split subsets, SplitLight analyzes core and temporal dataset statistics, characterizes repeat consumption patterns and timestamp anomalies, and diagnoses split validity, including temporal leakage, cold-user/item exposure, and distribution shifts. SplitLight further allows side-by-side comparison of alternative splitting strategies through comprehensive aggregated summaries and interactive visualizations. Delivered as both a Python toolkit and an interactive no-code interface, SplitLight produces audit summaries that justify evaluation protocols and support transparent, reliable, and comparable experimentation in recommender systems research and industry.