# Take a Fresh Look at Recommender Systems from an Evaluation Standpoint

arXiv: https://arxiv.org/abs/2210.04149 · HTML: https://mlsanity.com/paper/2210.04149/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2022.

## Built on by (most cited first, 17 of 17)

- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Efficient user history modeling with amortized inference for deep learning recommendation models](https://mlsanity.com/paper/2412.06924/) (2024)
- [Offline Recommender System Evaluation under Unobserved Confounding](https://mlsanity.com/paper/2309.04222/) (2023)
- [CSRec: Rethinking Sequential Recommendation from A Causal Perspective](https://mlsanity.com/paper/2409.05872/) (2024)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [Recommendation Is a Dish Better Served Warm](https://mlsanity.com/paper/2508.07856/) (2025)
- [Impression-Aware Recommender Systems](https://mlsanity.com/paper/2308.07857/) (2023)
- [Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization](https://mlsanity.com/paper/2507.19473/) (2025)
- [Advancements in Recommender Systems: A Comprehensive Analysis Based on Data, Algorithms, and Evaluation](https://mlsanity.com/paper/2407.18937/) (2024)
- [Rolling Forward: Enhancing LightGCN with Causal Graph Convolution for Credit Bond Recommendation](https://mlsanity.com/paper/2503.14213/) (2025)
- [SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits](https://mlsanity.com/paper/2602.19339/) (2026)
- [HORIZON: A Benchmark for In-the-wild User Behaviour Modeling](https://mlsanity.com/paper/2604.17259/) (2026)
- [Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders](https://mlsanity.com/paper/2608.14021/) (2026)
- [Revisiting Generative vs. Dense Retrieval: A Static Value Prior Reverses the Gap](https://mlsanity.com/paper/2610.19065/) (2026)

## Abstract (arXiv)

Recommendation has become a prominent area of research in the field of Information Retrieval (IR). Evaluation is also a traditional research topic in this community. Motivated by a few counter-intuitive observations reported in recent studies, this perspectives paper takes a fresh look at recommender systems from an evaluation standpoint. Rather than examining metrics like recall, hit rate, or NDCG, or perspectives like novelty and diversity, the key focus here is on how these metrics are calculated when evaluating a recommender algorithm. Specifically, the commonly used train/test data splits and their consequences are re-examined. We begin by examining common data splitting methods, such as random split or leave-one-out, and discuss why the popularity baseline is poorly defined under such splits. We then move on to explore the two implications of neglecting a global timeline during evaluation: data leakage and oversimplification of user preference modeling. Afterwards, we present new perspectives on recommender systems, including techniques for evaluating algorithm performance that more accurately reflect real-world scenarios, and possible approaches to consider decision contexts in user preference modeling.