# Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders

arXiv: https://arxiv.org/abs/2308.06885 · HTML: https://mlsanity.com/paper/2308.06885/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems](https://mlsanity.com/paper/1905.04413/) (2019)
- [Embarrassingly Shallow Autoencoders for Sparse Data](https://mlsanity.com/paper/1905.03375/) (2019)
- [Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking](https://mlsanity.com/paper/1707.05176/) (2017)
- [Sequential Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1811.09975/) (2018)
- [Trust your neighbors: A comprehensive survey of neighborhood-based methods for recommender systems](https://mlsanity.com/paper/2109.04584/) (2021)
- [Beyond NDCG: behavioral testing of recommender systems with RecList](https://mlsanity.com/paper/2111.09963/) (2021)
- [Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce](https://mlsanity.com/paper/1809.03186/) (2018)
- [Orthogonal Inductive Matrix Completion](https://mlsanity.com/paper/2004.01653/) (2020)
- [Evaluation of recommender systems in streaming environments](https://mlsanity.com/paper/1504.08175/) (2015)
- [EvalRS 2023. Well-Rounded Recommender Systems For Real-World Deployments](https://mlsanity.com/paper/2304.07145/) (2023)

## Built on by (most cited first, 1 of 1)

- [Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems](https://mlsanity.com/paper/2507.09566/) (2025)

## Abstract (arXiv)

The evaluation of recommendation systems is a complex task. The offline and online evaluation metrics for recommender systems are ambiguous in their true objectives. The majority of recently published papers benchmark their methods using ill-posed offline evaluation methodology that often fails to predict true online performance. Because of this, the impact that academic research has on the industry is reduced. The aim of our research is to investigate and compare the online performance of offline evaluation metrics. We show that penalizing popular items and considering the time of transactions during the evaluation significantly improves our ability to choose the best recommendation model for a live recommender system. Our results, averaged over five large-size real-world live data procured from recommenders, aim to help the academic community to understand better offline evaluation and optimization criteria that are more relevant for real applications of recommender systems.