# DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation

arXiv: https://arxiv.org/abs/2206.10848 · HTML: https://mlsanity.com/paper/2206.10848/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 200 references that are in the map.
- First posted on arXiv 22 Jun 2022.

## Builds on (most cited first, 30 of 61)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Neural Factorization Machines for Sparse Predictive Analytics](https://mlsanity.com/paper/1708.05027/) (2017)
- [RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems](https://mlsanity.com/paper/1803.03467/) (2018)
- [RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms](https://mlsanity.com/paper/2011.01731/) (2020)
- [Knowledge Graph Convolutional Networks for Recommender Systems](https://mlsanity.com/paper/1904.12575/) (2019)
- [Disentangled Graph Collaborative Filtering](https://mlsanity.com/paper/2007.01764/) (2020)
- [Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems](https://mlsanity.com/paper/1905.04413/) (2019)
- [Fast Matrix Factorization for Online Recommendation with Implicit Feedback](https://mlsanity.com/paper/1708.05024/) (2017)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences](https://mlsanity.com/paper/1902.06236/) (2019)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation](https://mlsanity.com/paper/1901.08907/) (2019)
- [Improving Question Answering by Commonsense-Based Pre-Training](https://mlsanity.com/paper/1809.03568/) (2018)
- [RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback](https://mlsanity.com/paper/1912.11160/) (2019)
- [Adversarial Personalized Ranking for Recommendation](https://mlsanity.com/paper/1808.03908/) (2018)
- [On the Difficulty of Evaluating Baselines: A Study on Recommender Systems](https://mlsanity.com/paper/1905.01395/) (2019)
- [Jointly Learning Explainable Rules for Recommendation with Knowledge Graph](https://mlsanity.com/paper/1903.03714/) (2019)
- [Spectral Collaborative Filtering](https://mlsanity.com/paper/1808.10523/) (2018)
- [Learning to Hash with Graph Neural Networks for Recommender Systems](https://mlsanity.com/paper/2003.01917/) (2020)
- [Collaborative Memory Network for Recommendation Systems](https://mlsanity.com/paper/1804.10862/) (2018)
- [Joint Item Recommendation and Attribute Inference: An Adaptive Graph Convolutional Network Approach](https://mlsanity.com/paper/2005.12021/) (2020)
- [Attentional Graph Convolutional Networks for Knowledge Concept Recommendation in MOOCs in a Heterogeneous View](https://mlsanity.com/paper/2006.13257/) (2020)
- [From Zero-Shot Learning to Cold-Start Recommendation](https://mlsanity.com/paper/1906.08511/) (2019)
- [An Efficient Neighborhood-based Interaction Model for Recommendation on Heterogeneous Graph](https://mlsanity.com/paper/2007.00216/) (2020)
- [DE-RRD: A Knowledge Distillation Framework for Recommender System](https://mlsanity.com/paper/2012.04357/) (2020)

## Built on by (most cited first, 19 of 19)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Large Language Models for Intent-Driven Session Recommendations](https://mlsanity.com/paper/2312.07552/) (2023)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Adaptive In-Context Learning with Large Language Models for Bundle Generation](https://mlsanity.com/paper/2312.16262/) (2023)
- [Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation](https://mlsanity.com/paper/2403.16427/) (2024)
- [Active Large Language Model-based Knowledge Distillation for Session-based Recommendation](https://mlsanity.com/paper/2502.15685/) (2024)
- [When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?](https://mlsanity.com/paper/2305.01801/) (2023)
- [A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems](https://mlsanity.com/paper/2406.17335/) (2024)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [Uncertain Multi-Objective Recommendation via Orthogonal Meta-Learning Enhanced Bayesian Optimization](https://mlsanity.com/paper/2502.13180/) (2025)
- [Cross-domain Transfer of Valence Preferences via a Meta-optimization Approach](https://mlsanity.com/paper/2406.16494/) (2024)
- [LightKG: Efficient Knowledge-Aware Recommendations with Simplified GNN Architecture](https://mlsanity.com/paper/2506.10347/) (2025)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [Eval4DiRec: A Unified and Systematic Evaluation Framework for Diffusion-based Recommender Systems](https://mlsanity.com/paper/2609.34404/) (2026)
- [What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders](https://mlsanity.com/paper/2401.11632/) (2024)
- [Towards Fair and Rigorous Evaluations: Hyperparameter Optimization for Top-N Recommendation Task with Implicit Feedback](https://mlsanity.com/paper/2408.07630/) (2024)
- [Modeling User Preferences as Distributions for Optimal Transport-Based Cross-Domain Recommendation under Non-Overlapping Settings](https://mlsanity.com/paper/2508.16210/) (2025)
- [A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2508.19620/) (2025)

## Abstract (arXiv)

Recently, one critical issue looms large in the field of recommender systems -- there are no effective benchmarks for rigorous evaluation -- which consequently leads to unreproducible evaluation and unfair comparison. We, therefore, conduct studies from the perspectives of practical theory and experiments, aiming at benchmarking recommendation for rigorous evaluation. Regarding the theoretical study, a series of hyper-factors affecting recommendation performance throughout the whole evaluation chain are systematically summarized and analyzed via an exhaustive review on 141 papers published at eight top-tier conferences within 2017-2020. We then classify them into model-independent and model-dependent hyper-factors, and different modes of rigorous evaluation are defined and discussed in-depth accordingly. For the experimental study, we release DaisyRec 2.0 library by integrating these hyper-factors to perform rigorous evaluation, whereby a holistic empirical study is conducted to unveil the impacts of different hyper-factors on recommendation performance. Supported by the theoretical and experimental studies, we finally create benchmarks for rigorous evaluation by proposing standardized procedures and providing performance of ten state-of-the-arts across six evaluation metrics on six datasets as a reference for later study. Overall, our work sheds light on the issues in recommendation evaluation, provides potential solutions for rigorous evaluation, and lays foundation for further investigation.