mlsanityData to 14 Oct 2026

On the Difficulty of Evaluating Baselines: A Study on Recommender Systems

arXiv 1905.01395 ↗First posted 4 May 2019cs.IR cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 8; 2023: 13; 2024: 5; 2025: 5; 2026: 1.

46 papers.

Sort:

By area:

  1. 2021A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendationcited by 59
  2. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  3. 2022Preference Dynamics Under Personalized Recommendationscited by 25
  4. 2021Revisiting the Performance of iALS on Item Recommendation Benchmarkscited by 21
  5. 2022DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluationcited by 19
  6. 2021Item Recommendation from Implicit Feedbackcited by 18
  7. 2021Towards a Better Understanding of Linear Models for Recommendationcited by 18
  8. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17
  9. 2021User Response Prediction in Online Advertisingcited by 15
  10. 2021On component interactions in two-stage recommender systemscited by 14
  11. 2023Scaling Laws Do Not Scalecited by 13
  12. 2023On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendationcited by 13
34 more
  1. 2021Private Alternating Least Squares: Practical Private Matrix Completion with Tighter Ratescited by 10
  2. 2021On Estimating Recommendation Evaluation Metrics under Samplingcited by 9
  3. 2021Quantifying Availability and Discovery in Recommender Systems via Stochastic Reachabilitycited by 8
  4. 2023It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendationcited by 7
  5. 2023RecFusion: A Binomial Diffusion Process for 1D Data for Recommendationcited by 7
  6. 2023Private Learning with Public Featurescited by 6
  7. 2025ToolSpectrum : Towards Personalized Tool Utilization for Large Language Modelscited by 6
  8. 2023Revisiting Recommendation Loss Functions through Contrastive Learning (Technical Report)cited by 5
  9. 2025Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Argumentscited by 5
  10. 2021Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systemscited by 4
  11. 2023When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?cited by 4
  12. 2024A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systemscited by 4
  13. 2024Questionable practices in machine learningcited by 4
  14. 2021Local Search Algorithms for Rank-Constrained Convex Optimizationcited by 3
  15. 2022Factorized Structured Regression for Large-Scale Varying Coefficient Modelscited by 3
  16. 2021ALX: Large Scale Matrix Factorization on TPUscited by 2
  17. 2023A Lightweight Method for Modeling Confidence in Recommendations with Learned Beta Distributionscited by 2
  18. 2023Impression-Informed Multi-Behavior Recommender System: A Hierarchical Graph Attention Approachcited by 2
  19. 2024Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Needcited by 2
  20. 2022Efficient Mixed Dimension Embeddings for Matrix Factorizationcited by 1
  21. 2022Where Do We Go From Here? Guidelines For Offline Recommender Evaluationcited by 1
  22. 2022Symphony in the Latent Space: Provably Integrating High-dimensional Techniques with Non-linear Machine Learning Modelscited by 1
  23. 2023Probabilistic Unrolling: Scalable, Inverse-Free Maximum Likelihood Estimation for Latent Gaussian Modelscited by 1
  24. 2023Of Spiky SVDs and Music Recommendationcited by 1
  25. 2023Performance Comparison of Session-based Recommendation Algorithms based on GNNscited by 1
  26. 2025ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Testscited by 1
  27. 2021UserReg: A Simple but Strong Model for Rating Predictioncited by 0
  28. 2021Simulation comparisons between Bayesian and de-biased estimators in low-rank matrix completioncited by 0
  29. 2022The effectiveness of factorization and similarity blendingcited by 0
  30. 2023(Debiased) Contrastive Learning Loss for Recommendation (Technical Report)cited by 0
  31. 2024Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression modelscited by 0
  32. 2024Greedy SLIM: A SLIM-Based Approach For Preference Elicitationcited by 0
  33. 2025An Interdisciplinary and Cross-Task Review on Missing Data Imputationcited by 0
  34. 2026On the Regularization Landscape for the Linear Recommendation Modelscited by 0