# Do Offline Metrics Predict Online Performance in Recommender Systems?

arXiv: https://arxiv.org/abs/2011.07931 · HTML: https://mlsanity.com/paper/2011.07931/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2020.

## Built on by (most cited first, 25 of 25)

- [Modeling Content Creator Incentives on Algorithm-Curated Platforms](https://mlsanity.com/paper/2206.13102/) (2022)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)
- [Fair ranking: a critical review, challenges, and future directions](https://mlsanity.com/paper/2201.12662/) (2022)
- [On component interactions in two-stage recommender systems](https://mlsanity.com/paper/2106.14979/) (2021)
- [Breaking Feedback Loops in Recommender Systems with Causal Inference](https://mlsanity.com/paper/2207.01616/) (2022)
- [Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression](https://mlsanity.com/paper/2310.11428/) (2023)
- [T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems](https://mlsanity.com/paper/2107.08959/) (2021)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [The Stereotyping Problem in Collaboratively Filtered Recommender Systems](https://mlsanity.com/paper/2106.12622/) (2021)
- [Quantifying Availability and Discovery in Recommender Systems via Stochastic Reachability](https://mlsanity.com/paper/2107.00833/) (2021)
- [Towards Psychologically-Grounded Dynamic Preference Models](https://mlsanity.com/paper/2208.01534/) (2022)
- [In the Eye of the Beholder: Robust Prediction with Causal User Modeling](https://mlsanity.com/paper/2206.00416/) (2022)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [Algorithmic Censoring in Dynamic Learning Systems](https://mlsanity.com/paper/2305.09035/) (2023)
- [Harm Mitigation in Recommender Systems under User Preference Dynamics](https://mlsanity.com/paper/2406.09882/) (2024)
- [When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?](https://mlsanity.com/paper/2305.01801/) (2023)
- [Learning to Suggest Breaks: Sustainable Optimization of Long-Term User Engagement](https://mlsanity.com/paper/2211.13585/) (2022)
- [Counterfactual Augmentation for Multimodal Learning Under Presentation Bias](https://mlsanity.com/paper/2305.14083/) (2023)
- [Simulating News Recommendation Ecosystem for Fun and Profit](https://mlsanity.com/paper/2305.14103/) (2023)
- [Review of Explainable Graph-Based Recommender Systems](https://mlsanity.com/paper/2408.00166/) (2024)
- [Contextual Bandits with Sparse Data in Web setting](https://mlsanity.com/paper/2105.02873/) (2021)
- [Cross-Dataset Propensity Estimation for Debiasing Recommender Systems](https://mlsanity.com/paper/2212.13892/) (2022)
- [DPR: An Algorithm Mitigate Bias Accumulation in Recommendation feedback loops](https://mlsanity.com/paper/2311.05864/) (2023)
- [Enhancing Live Broadcast Engagement: A Multi-modal Approach to Short Video Recommendations Using MMGCN and User Preferences](https://mlsanity.com/paper/2506.23085/) (2025)
- [Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships](https://mlsanity.com/paper/2603.07671/) (2026)

## Abstract (arXiv)

Recommender systems operate in an inherently dynamical setting. Past recommendations influence future behavior, including which data points are observed and how user preferences change. However, experimenting in production systems with real user dynamics is often infeasible, and existing simulation-based approaches have limited scale. As a result, many state-of-the-art algorithms are designed to solve supervised learning problems, and progress is judged only by offline metrics. In this work we investigate the extent to which offline metrics predict online performance by evaluating eleven recommenders across six controlled simulated environments. We observe that offline metrics are correlated with online performance over a range of environments. However, improvements in offline metrics lead to diminishing returns in online performance. Furthermore, we observe that the ranking of recommenders varies depending on the amount of initial offline data available. We study the impact of adding exploration strategies, and observe that their effectiveness, when compared to greedy recommendation, is highly dependent on the recommendation algorithm. We provide the environments and recommenders described in this paper as Reclab: an extensible ready-to-use simulation framework at https://github.com/berkeley-reclab/RecLab.