# Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce

arXiv: https://arxiv.org/abs/1809.03186 · HTML: https://mlsanity.com/paper/1809.03186/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2018.

## Built on by (most cited first, 6 of 6)

- [Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment](https://mlsanity.com/paper/2210.13005/) (2022)
- [MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning](https://mlsanity.com/paper/2102.11448/) (2021)
- [Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders](https://mlsanity.com/paper/2308.06885/) (2023)
- [Online and Offline Evaluations of Collaborative Filtering and Content Based Recommender Systems](https://mlsanity.com/paper/2411.01354/) (2024)
- [Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems](https://mlsanity.com/paper/2507.09566/) (2025)
- [LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising](https://mlsanity.com/paper/2610.14550/) (2026)

## Abstract (arXiv)

In this paper, we present our work towards comparing on-line and off-line evaluation metrics in the context of small e-commerce recommender systems. Recommending on small e-commerce enterprises is rather challenging due to the lower volume of interactions and low user loyalty, rarely extending beyond a single session. On the other hand, we usually have to deal with lower volumes of objects, which are easier to discover by users through various browsing/searching GUIs. The main goal of this paper is to determine applicability of off-line evaluation metrics in learning true usability of recommender systems (evaluated on-line in A/B testing). In total 800 variants of recommending algorithms were evaluated off-line w.r.t. 18 metrics covering rating-based, ranking-based, novelty and diversity evaluation. The off-line results were afterwards compared with on-line evaluation of 12 selected recommender variants and based on the results, we tried to learn and utilize an off-line to on-line results prediction model. Off-line results shown a great variance in performance w.r.t. different metrics with the Pareto front covering 68% of the approaches. Furthermore, we observed that on-line results are considerably affected by the novelty of users. On-line metrics correlates positively with ranking-based metrics (AUC, MRR, nDCG) for novice users, while too high values of diversity and novelty had a negative impact on the on-line results for them. For users with more visited items, however, the diversity became more important, while ranking-based metrics relevance gradually decrease.