mlsanityData to 7 Oct 2026

Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations

arXiv 2408.12008 ↗First posted 21 Aug 2024cs.IR cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

18 of 30 references.

References · 18

  1. 2018Self-Attentive Sequential Recommendationcited by 1,169
  2. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  3. 2015Image-based Recommendations on Styles and Substitutescited by 702
  4. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  5. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 473
  6. 2016Yelp Dataset Challenge: Review Rating Predictioncited by 84
  7. 2018Sequence-Aware Recommender Systemscited by 58
  8. 2020Exploring Data Splitting Strategies for the Evaluation of Recommendation Modelscited by 48
  9. 2021Non-invasive Self-attention for Side Information Fusion in Sequential Recommendationcited by 39
  10. 2020A Critical Study on Data Leakage in Recommender System Offline Evaluationcited by 38
  11. 2023Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?cited by 30
  12. 2022A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendationcited by 22
6 more
  1. 2023Widespread Flaws in Offline Evaluation of Recommender Systemscited by 21
  2. 2022Rank List Sensitivity of Recommender Systems to Interaction Perturbationscited by 20
  3. 2022Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?cited by 17
  4. 2022Take a Fresh Look at Recommender Systems from an Evaluation Standpointcited by 16
  5. 2024From Variability to Stability: Advancing RecSys Benchmarking Practicescited by 12
  6. 2023Investigating the Robustness of Sequential Recommender Systems Against Training Data Perturbationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 6.

15 papers.

Sort:

  1. 2025Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenderscited by 12
  2. 2025Lost in Sequence: Do Large Language Models Understand Sequential Recommendation?cited by 8
  3. 2024RePlay: a Recommendation Framework for Experimentation and Production Usecited by 3
  4. 2025Ultra Fast Warm Start Solution for Graph Recommendationscited by 2
  5. 2025Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initializationcited by 1
  6. 2024Stalactite: Toolbox for Fast Prototyping of Vertical Federated Learning Systemscited by 0
  7. 2025Looking around you: external information enhances representations for event sequencescited by 0
  8. 2025Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervisioncited by 0
  9. 2025How to model Human Actions distribution with Event Sequence Datacited by 0
  10. 2026SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splitscited by 0
  11. 2026Cross-Representation Knowledge Transfer for Improved Sequential Recommendationscited by 0
  12. 2026HORIZON: A Benchmark for In-the-wild User Behaviour Modelingcited by 0
3 more
  1. 2026Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillationcited by 0
  2. 2026Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomiescited by 0
  3. 2026Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenderscited by 0