# Widespread Flaws in Offline Evaluation of Recommender Systems

arXiv: https://arxiv.org/abs/2307.14951 · HTML: https://mlsanity.com/paper/2307.14951/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2023.

## Built on by (most cited first, 22 of 22)

- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Autoregressive Generation Strategies for Top-K Sequential Recommendations](https://mlsanity.com/paper/2409.17730/) (2024)
- [Evaluating Performance and Bias of Negative Sampling in Large-Scale Sequential Recommendation Models](https://mlsanity.com/paper/2410.17276/) (2024)
- [Building a Scalable, Effective, and Steerable Search and Ranking Platform](https://mlsanity.com/paper/2409.02856/) (2024)
- [eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion](https://mlsanity.com/paper/2508.06450/) (2025)
- [Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters](https://mlsanity.com/paper/2411.16645/) (2024)
- [Are We Really Measuring Progress? Transferring Insights from Evaluating Recommender Systems to Temporal Link Prediction](https://mlsanity.com/paper/2506.12588/) (2025)
- [Recommendation Is a Dish Better Served Warm](https://mlsanity.com/paper/2508.07856/) (2025)
- [Let It Go? Not Quite: Addressing Item Cold Start in Sequential Recommendations with Content-Based Initialization](https://mlsanity.com/paper/2507.19473/) (2025)
- [Balancing Accuracy and Novelty with Sub-Item Popularity](https://mlsanity.com/paper/2508.05198/) (2025)
- [We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Later](https://mlsanity.com/paper/2509.09414/) (2025)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [ProfiLLM: Utility-Aligned Agentic User Profiling for Industrial Ride-Hailing Dispatch](https://mlsanity.com/paper/2606.18803/) (2026)
- [Compositions of Variant Experts for Integrating Short-Term and Long-Term Preferences](https://mlsanity.com/paper/2506.23170/) (2025)
- [Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems](https://mlsanity.com/paper/2507.09566/) (2025)
- [Semantic IDs for Music Recommendation](https://mlsanity.com/paper/2507.18800/) (2025)
- [Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank](https://mlsanity.com/paper/2507.20753/) (2025)
- [SplitLight: An Exploratory Toolkit for Recommender Systems Datasets and Splits](https://mlsanity.com/paper/2602.19339/) (2026)
- [Position: Generative Engine Optimization Creates Underexamined Risks, Governance Must Target Concentration, Disclosure, and Academic Blind Spots](https://mlsanity.com/paper/2606.12439/) (2026)
- [Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders](https://mlsanity.com/paper/2608.14021/) (2026)
- [Revisiting Generative vs. Dense Retrieval: A Static Value Prior Reverses the Gap](https://mlsanity.com/paper/2610.19065/) (2026)

## Abstract (arXiv)

Even though offline evaluation is just an imperfect proxy of online performance -- due to the interactive nature of recommenders -- it will probably remain the primary way of evaluation in recommender systems research for the foreseeable future, since the proprietary nature of production recommenders prevents independent validation of A/B test setups and verification of online results. Therefore, it is imperative that offline evaluation setups are as realistic and as flawless as they can be. Unfortunately, evaluation flaws are quite common in recommender systems research nowadays, due to later works copying flawed evaluation setups from their predecessors without questioning their validity. In the hope of improving the quality of offline evaluation of recommender systems, we discuss four of these widespread flaws and why researchers should avoid them.