# Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

arXiv: https://arxiv.org/abs/2103.02590 · HTML: https://mlsanity.com/paper/2103.02590/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2021.

## Built on by (most cited first, 28 of 28)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Can Small Language Models be Good Reasoners for Sequential Recommendation?](https://mlsanity.com/paper/2403.04260/) (2024)
- [Evaluating ChatGPT as a Recommender System: A Rigorous Approach](https://mlsanity.com/paper/2309.03613/) (2023)
- [DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation](https://mlsanity.com/paper/2206.10848/) (2022)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [Reenvisioning Collaborative Filtering vs Matrix Factorization](https://mlsanity.com/paper/2107.13472/) (2021)
- [When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?](https://mlsanity.com/paper/2305.01801/) (2023)
- [A Catalog of Fairness-Aware Practices in Machine Learning Engineering](https://mlsanity.com/paper/2408.16683/) (2024)
- [Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?](https://mlsanity.com/paper/2307.09985/) (2023)
- [Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis](https://mlsanity.com/paper/2308.00404/) (2023)
- [RePlay: a Recommendation Framework for Experimentation and Production Use](https://mlsanity.com/paper/2409.07272/) (2024)
- [Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation](https://mlsanity.com/paper/2508.04571/) (2025)
- [Evaluating the performance-deviation of itemKNN in RecBole and LensKit](https://mlsanity.com/paper/2407.13531/) (2024)
- [The Tsetlin Machine Goes Deep: Logical Learning and Reasoning With Graphs](https://mlsanity.com/paper/2507.14874/) (2025)
- [Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools](https://mlsanity.com/paper/2201.06444/) (2022)
- [Where Do We Go From Here? Guidelines For Offline Recommender Evaluation](https://mlsanity.com/paper/2211.01261/) (2022)
- [MOReGIn: Multi-Objective Recommendation at the Global and Individual Levels](https://mlsanity.com/paper/2401.12593/) (2024)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [Quantifying User Coherence: A Unified Framework for Analyzing Recommender Systems Across Domains](https://mlsanity.com/paper/2410.02453/) (2024)
- [ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)](https://mlsanity.com/paper/2503.16661/) (2025)
- [Universal Reusability in Recommender Systems: The Case for Dataset- and Task-Independent Frameworks](https://mlsanity.com/paper/2506.03391/) (2025)
- [Bias vs Bias -- Dawn of Justice: A Fair Fight in Recommendation Systems](https://mlsanity.com/paper/2506.18327/) (2025)
- [Parameter-Efficient Single Collaborative Branch for Recommendation](https://mlsanity.com/paper/2508.03518/) (2025)
- [Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives](https://mlsanity.com/paper/2512.01179/) (2025)
- [Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations](https://mlsanity.com/paper/2601.01997/) (2026)
- [SIDScope: A Diagnostic Resource for Semantic-ID Interfaces in Generative Recommendation](https://mlsanity.com/paper/2608.18779/) (2026)
- [AutoRecLab: Describe the Experiment, Get the Code!](https://mlsanity.com/paper/2609.21863/) (2026)

## Abstract (arXiv)

Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test). The aim is to provide the researchers with a tool to ease (and make them reproducible) all the experimental evaluation phases, from data reading to results collection. Elliot is available on GitHub (https://github.com/sisinflab/elliot).