mlsanityData to 14 Oct 2026

SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models

arXiv 2604.09553 ↗First posted 31 Jan 2026cs.IR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM development has aroused great interest in Sequential Recommendation (SR) applications. However, comprehensive evaluation of SR models remains lacking due to the limitations of the existing benchmarks: 1) an overemphasis on accuracy, ignoring other real-world demands (e.g., fairness); 2) existing datasets fail to unleash LLMs' potential, leading to unfair comparison between Neural-Network-based SR (NN-SR) models and LLM-based SR (LLM-SR) models; and 3) no reliable mechanism for extracting task-specific answers from unstructured LLM outputs. To address these limitations, we propose SRBench, a comprehensive SR benchmark with three core designs: 1) a multi-dimensional framework covering accuracy, fairness, stability and efficiency, aligned with practical demands; 2) a unified input paradigm via prompt engineering to boost LLM-SR performance and enable fair comparisons between models; 3) a novel prompt-extractor-coupled extraction mechanism, which captures answers from LLM outputs through prompt-enforced output formatting and a numeric-oriented extractor. We have used SRBench to evaluate 13 mainstream models and discovered some meaningful insights (e.g., LLM-SR models overfocus on item popularity but lack deep understanding of item quality). Concisely, SRBench enables fair and comprehensive assessments for SR models, underpinning future research and practical application.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

35 of 54 references.

References · 35

  1. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
  2. 2017Neural Collaborative Filteringcited by 1,111
  3. 2019Neural Graph Collaborative Filteringcited by 661
  4. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 476
  5. 2018Assessing Generative Models via Precision and Recallcited by 406
  6. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
  7. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  8. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 242
  9. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 193
  10. 2017Recurrent Neural Networks with Top-k Gains for Session-based Recommendationscited by 165
  11. 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 157
  12. 2018A Simple Convolutional Generative Network for Next Item Recommendationcited by 120
23 more
  1. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 114
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  3. 2021Sequential Recommendation with Graph Neural Networkscited by 89
  4. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  5. 2023Towards Open-World Recommendation with Knowledge Augmentation from Large Language Modelscited by 74
  6. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  7. 2021Dynamic Graph Neural Networks for Sequential Recommendationcited by 57
  8. 2023Zero-Shot Next-Item Recommendation using Large Pretrained Language Modelscited by 53
  9. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 53
  10. 2019CosRec: 2D Convolutional Neural Networks for Sequential Recommendationcited by 37
  11. 2023LLMRec: Benchmarking Large Language Models on Recommendation Taskcited by 30
  12. 2024Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recencycited by 28
  13. 2018User Fairness in Recommender Systemscited by 21
  14. 2023OpenP5: An Open-Source Platform for Developing, Training, and Evaluating LLM-based Recommender Systemscited by 19
  15. 2024Beyond Utility: Evaluating LLM as Recommendercited by 15
  16. 2023Large Language Models are Not Stable Recommender Systemscited by 13
  17. 2023A Comprehensive Survey of Evaluation Techniques for Recommendation Systemscited by 12
  18. 2022DDGHM: Dual Dynamic Graph with Hybrid Metric Training for Cross-Domain Sequential Recommendationcited by 11
  19. 2023Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Studycited by 5
  20. 2024Improving LLMs for Recommendation with Out-Of-Vocabulary Tokenscited by 5
  21. 2024A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender Systemcited by 4
  22. 2025Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendationcited by 4
  23. 2025Improving the Performance of Sequential Recommendation Systems with an Extended Large Language Modelcited by 1

Built on by

Not cited yet.