# A Comprehensive Survey of Evaluation Techniques for Recommendation Systems

arXiv: https://arxiv.org/abs/2312.16015 · HTML: https://mlsanity.com/paper/2312.16015/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 49 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 9 of 9)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecasting](https://mlsanity.com/paper/2211.02989/) (2022)
- [An Overview of Deep Learning Architectures in Few-Shot Learning Domain](https://mlsanity.com/paper/2008.06365/) (2020)
- [Microsoft Recommenders: Tools to Accelerate Developing Recommender Systems](https://mlsanity.com/paper/2008.13528/) (2020)
- [Leveraging Generative AI Models for Synthetic Data Generation in Healthcare: Balancing Research and Privacy](https://mlsanity.com/paper/2305.05247/) (2023)
- [Auto-labelling of Bug Report using Natural Language Processing](https://mlsanity.com/paper/2212.06334/) (2022)
- [A Comparative Study of Text Embedding Models for Semantic Text Similarity in Bug Reports](https://mlsanity.com/paper/2308.09193/) (2023)
- [Global Message Ordering using Distributed Kafka Clusters](https://mlsanity.com/paper/2309.04918/) (2023)

## Built on by (most cited first, 12 of 12)

- [Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework](https://mlsanity.com/paper/2406.14783/) (2024)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [Intersectional Unfairness Discovery](https://mlsanity.com/paper/2405.20790/) (2024)
- [Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Models](https://mlsanity.com/paper/2502.15854/) (2025)
- [Privacy-Utility-Bias Trade-offs for Privacy-Preserving Recommender Systems](https://mlsanity.com/paper/2511.22515/) (2025)
- [Bayesian-Guided Diversity in Sequential Sampling for Recommender Systems](https://mlsanity.com/paper/2506.21617/) (2025)
- [Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation](https://mlsanity.com/paper/2511.16478/) (2025)
- [A Framework for Mining Collectively-Behaving Bots in MMORPGs](https://mlsanity.com/paper/2501.10461/) (2025)
- [Hier-COS: Making Deep Features Hierarchy-aware via Composition of Orthogonal Subspaces](https://mlsanity.com/paper/2503.07853/) (2025)
- [Model-agnostic post-hoc explainability for recommender systems](https://mlsanity.com/paper/2509.10245/) (2025)
- [SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2604.09553/) (2026)
- [Can We Steer the Black-Box? Towards Controllability-Centric Evaluation of Recommender Systems with Collaborative Agents](https://mlsanity.com/paper/2607.13418/) (2026)

## Abstract (arXiv)

The effectiveness of recommendation systems is pivotal to user engagement and satisfaction in online platforms. As these recommendation systems increasingly influence user choices, their evaluation transcends mere technical performance and becomes central to business success. This paper addresses the multifaceted nature of recommendations system evaluation by introducing a comprehensive suite of metrics, each tailored to capture a distinct aspect of system performance. We discuss * Similarity Metrics: to quantify the precision of content-based filtering mechanisms and assess the accuracy of collaborative filtering techniques. * Candidate Generation Metrics: to evaluate how effectively the system identifies a broad yet relevant range of items. * Predictive Metrics: to assess the accuracy of forecasted user preferences. * Ranking Metrics: to evaluate the effectiveness of the order in which recommendations are presented. * Business Metrics: to align the performance of the recommendation system with economic objectives. Our approach emphasizes the contextual application of these metrics and their interdependencies. In this paper, we identify the strengths and limitations of current evaluation practices and highlight the nuanced trade-offs that emerge when optimizing recommendation systems across different metrics. The paper concludes by proposing a framework for selecting and interpreting these metrics to not only improve system performance but also to advance business goals. This work is to aid researchers and practitioners in critically assessing recommendation systems and fosters the development of more nuanced, effective, and economically viable personalization strategies. Our code is available at GitHub - https://github.com/aryan-jadon/Evaluation-Metrics-for-Recommendation-Systems.