# Evaluation Metrics for Item Recommendation under Sampling

arXiv: https://arxiv.org/abs/1912.02263 · HTML: https://mlsanity.com/paper/1912.02263/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2019.

## Built on by (most cited first, 6 of 6)

- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [On Estimating Recommendation Evaluation Metrics under Sampling](https://mlsanity.com/paper/2103.01474/) (2021)
- [Rethinking Neural vs. Matrix-Factorization Collaborative Filtering: the Theoretical Perspectives](https://mlsanity.com/paper/2110.12141/) (2021)
- [Intrinsic Self-Supervision for Data Quality Audits](https://mlsanity.com/paper/2305.17048/) (2023)
- [InDiReCT: Language-Guided Zero-Shot Deep Metric Learning for Images](https://mlsanity.com/paper/2211.12760/) (2022)
- [Efficient Data-specific Model Search for Collaborative Filtering](https://mlsanity.com/paper/2106.07453/) (2021)

## Abstract (arXiv)

The task of item recommendation requires ranking a large catalogue of items given a context. Item recommendation algorithms are evaluated using ranking metrics that depend on the positions of relevant items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates sampled metrics in more detail and shows that sampled metrics are inconsistent with their exact version. Sampled metrics do not persist relative statements, e.g., 'algorithm A is better than B', not even in expectation. Moreover the smaller the sampling size, the less difference between metrics, and for very small sampling size, all metrics collapse to the AUC metric.