# On the Generalizability and Predictability of Recommender Systems

arXiv: https://arxiv.org/abs/2206.11886 · HTML: https://mlsanity.com/paper/2206.11886/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 101 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Embarrassingly Shallow Autoencoders for Sparse Data](https://mlsanity.com/paper/1905.03375/) (2019)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research](https://mlsanity.com/paper/1911.07698/) (2019)
- [Generating Personalized Recipes from Historical User Preferences](https://mlsanity.com/paper/1909.00105/) (2019)
- [Understanding Echo Chambers in E-commerce Recommender Systems](https://mlsanity.com/paper/2007.02474/) (2020)
- [Spectral Collaborative Filtering](https://mlsanity.com/paper/1808.10523/) (2018)
- [Applying Deep Learning To Airbnb Search](https://mlsanity.com/paper/1810.09591/) (2018)
- [Addressing Marketing Bias in Product Recommendations](https://mlsanity.com/paper/1912.01799/) (2019)
- [Frappe: Understanding the Usage and Perception of Mobile App Recommendations In-The-Wild](https://mlsanity.com/paper/1505.03014/) (2015)
- [Sequential Evaluation and Generation Framework for Combinatorial Recommender System](https://mlsanity.com/paper/1902.00245/) (2019)
- [Auto-Surprise: An Automated Recommender-System (AutoRecSys) Library with Tree of Parzens Estimator (TPE) Optimization](https://mlsanity.com/paper/2008.13532/) (2020)
- [AutoRec: An Automated Recommender System](https://mlsanity.com/paper/2007.07224/) (2020)
- [One-at-a-time: A Meta-Learning Recommender-System for Recommendation-Algorithm Selection on Micro Level](https://mlsanity.com/paper/1805.12118/) (2018)
- [CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering](https://mlsanity.com/paper/1803.02250/) (2018)
- [Meta-Learned Per-Instance Algorithm Selection in Scholarly Recommender Systems](https://mlsanity.com/paper/1912.08694/) (2019)
- [NeuRec: On Nonlinear Transformation for Personalized Ranking](https://mlsanity.com/paper/1805.03002/) (2018)

## Built on by (most cited first, 4 of 4)

- [Measuring the Predictability of Recommender Systems using Structural Complexity Metrics](https://mlsanity.com/paper/2404.08829/) (2024)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [HORIZON: A Benchmark for In-the-wild User Behaviour Modeling](https://mlsanity.com/paper/2604.17259/) (2026)
- [Kernel-based External Validity of Experimental Studies](https://mlsanity.com/paper/2610.13625/) (2026)

## Abstract (arXiv)

While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, "how do we choose the right algorithm for a new dataset and performance metric?" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 18 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric, however, there are also strong correlations between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla.