# Online and Offline Evaluations of Collaborative Filtering and Content Based Recommender Systems

arXiv: https://arxiv.org/abs/2411.01354 · HTML: https://mlsanity.com/paper/2411.01354/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 9 of 9)

- [FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09433/) (2019)
- [Approximate Nearest Neighbor Search on High Dimensional Data --- Experiments, Analyses, and Improvement (v1.0)](https://mlsanity.com/paper/1610.02455/) (2016)
- [User-centered Evaluation of Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2103.06364/) (2021)
- [The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study](https://mlsanity.com/paper/1912.04696/) (2019)
- [Recommenders with a mission: assessing diversity in newsrecommendations](https://mlsanity.com/paper/2012.10185/) (2020)
- [Collaborative Similarity Embedding for Recommender Systems](https://mlsanity.com/paper/1902.06188/) (2019)
- [HybridSVD: When Collaborative Information is Not Enough](https://mlsanity.com/paper/1802.06398/) (2018)
- [Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce](https://mlsanity.com/paper/1809.03186/) (2018)
- [It's Time to Consider "Time" when Evaluating Recommender-System Algorithms [Proposal]](https://mlsanity.com/paper/1708.08447/) (2017)

## Abstract (arXiv)

Recommender systems are widely used AI applications designed to help users efficiently discover relevant items. The effectiveness of such systems is tied to the satisfaction of both users and providers. However, user satisfaction is complex and cannot be easily framed mathematically using information retrieval and accuracy metrics. While many studies evaluate accuracy through offline tests, a growing number of researchers argue that online evaluation methods such as A/B testing are better suited for this purpose. We have employed a variety of algorithms on different types of datasets divergent in size and subject, producing recommendations in various platforms, including media streaming services, digital publishing websites, e-commerce systems, and news broadcasting networks. Notably, our target websites and datasets are in Persian (Farsi) language. This study provides a comparative analysis of a large-scale recommender system that has been operating for the past year across about 70 websites in Iran, processing roughly 300 requests per second collectively. The system employs user-based and item-based recommendations using content-based, collaborative filtering, trend-based methods, and hybrid approaches. Through both offline and online evaluations, we aim to identify where these algorithms perform most efficiently and determine the best method for our specific needs, considering the dataset and system scale. Our methods of evaluation include manual evaluation, offline tests including accuracy and ranking metrics like hit-rate@k and nDCG, and online tests consisting of click-through rate (CTR). Additionally we analyzed and proposed methods to address cold-start and popularity bias.