# Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations

arXiv: https://arxiv.org/abs/2408.12008 · HTML: https://mlsanity.com/paper/2408.12008/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Sequence-Aware Recommender Systems](https://mlsanity.com/paper/1802.08452/) (2018)
- [Exploring Data Splitting Strategies for the Evaluation of Recommendation Models](https://mlsanity.com/paper/2007.13237/) (2020)
- [A Critical Study on Data Leakage in Recommender System Offline Evaluation](https://mlsanity.com/paper/2010.11060/) (2020)
- [Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation](https://mlsanity.com/paper/2103.03578/) (2021)
- [Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?](https://mlsanity.com/paper/2309.07602/) (2023)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [Widespread Flaws in Offline Evaluation of Recommender Systems](https://mlsanity.com/paper/2307.14951/) (2023)
- [Rank List Sensitivity of Recommender Systems to Interaction Perturbations](https://mlsanity.com/paper/2201.12686/) (2022)
- [Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?](https://mlsanity.com/paper/2206.12858/) (2022)
- [Take a Fresh Look at Recommender Systems from an Evaluation Standpoint](https://mlsanity.com/paper/2210.04149/) (2022)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Investigating the Robustness of Sequential Recommender Systems Against Training Data Perturbations](https://mlsanity.com/paper/2307.13165/) (2023)

## Built on by (most cited first, 15 of 15)

- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Lost in Sequence: Do Large Language Models Understand Sequential Recommendation?](https://mlsanity.com/paper/2502.13909/) (2025)
- [RePlay: a Recommendation Framework for Experimentation and Production Use](https://mlsanity.com/paper/2409.07272/) (2024)
- [Ultra Fast Warm Start Solution for Graph Recommendations](https://mlsanity.com/paper/2509.01549/) (2025)
- [Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization](https://mlsanity.com/paper/2507.19473/) (2025)
- [Stalactite: Toolbox for Fast Prototyping of Vertical Federated Learning Systems](https://mlsanity.com/paper/2409.15558/) (2024)
- [Looking around you: external information enhances representations for event sequences](https://mlsanity.com/paper/2502.10205/) (2025)
- [Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision](https://mlsanity.com/paper/2505.12526/) (2025)
- [How to model Human Actions distribution with Event Sequence Data](https://mlsanity.com/paper/2510.05856/) (2025)
- [SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits](https://mlsanity.com/paper/2602.19339/) (2026)
- [Cross-Representation Knowledge Transfer for Improved Sequential Recommendations](https://mlsanity.com/paper/2602.23471/) (2026)
- [HORIZON: A Benchmark for In-the-wild User Behaviour Modeling](https://mlsanity.com/paper/2604.17259/) (2026)
- [Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation](https://mlsanity.com/paper/2604.21536/) (2026)
- [Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies](https://mlsanity.com/paper/2606.07492/) (2026)
- [Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders](https://mlsanity.com/paper/2608.14021/) (2026)

## Abstract (arXiv)

Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.