# On Sampling Top-K Recommendation Evaluation

arXiv: https://arxiv.org/abs/2106.10621 · HTML: https://mlsanity.com/paper/2106.10621/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2021.

## Built on by (most cited first, 20 of 20)

- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2403.10557/) (2024)
- [On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendation](https://mlsanity.com/paper/2307.15053/) (2023)
- [Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems](https://mlsanity.com/paper/2508.05673/) (2025)
- [MobileRec: A Large-Scale Dataset for Mobile Apps Recommendation](https://mlsanity.com/paper/2303.06588/) (2023)
- [Priority Sampling of Large Language Models for Compilers](https://mlsanity.com/paper/2402.18734/) (2024)
- [Understand Legal Documents with Contextualized Large Language Models](https://mlsanity.com/paper/2303.12135/) (2023)
- [Revisiting Recommendation Loss Functions through Contrastive Learning (Technical Report)](https://mlsanity.com/paper/2312.08520/) (2023)
- [When Newer is Not Better: Does Deep Learning Really Benefit Recommendation From Implicit Feedback?](https://mlsanity.com/paper/2305.01801/) (2023)
- [Impression-Informed Multi-Behavior Recommender System: A Hierarchical Graph Attention Approach](https://mlsanity.com/paper/2309.03169/) (2023)
- [A Framework for Generating Conversational Recommendation Datasets from Behavioral Interactions](https://mlsanity.com/paper/2506.17285/) (2025)
- [Where Do We Go From Here? Guidelines For Offline Recommender Evaluation](https://mlsanity.com/paper/2211.01261/) (2022)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [(Debiased) Contrastive Learning Loss for Recommendation (Technical Report)](https://mlsanity.com/paper/2312.08517/) (2023)
- [Beyond Models! Explainable Data Valuation and Metric Adaption for Recommendation](https://mlsanity.com/paper/2502.08685/) (2025)
- [Gradients as an Action: Towards Communication-Efficient Federated Recommender Systems via Adaptive Action Sharing](https://mlsanity.com/paper/2507.08842/) (2025)
- [On the Reliability of Sampling Strategies in Offline Recommender Evaluation](https://mlsanity.com/paper/2508.05398/) (2025)
- [A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2508.19620/) (2025)
- [Empowering Denoising Sequential Recommendation with Large Language Model Embeddings](https://mlsanity.com/paper/2510.04239/) (2025)
- [Physics-Guided Sequence-Based Generative Framework for Acoustic Metamaterial Inverse Design](https://mlsanity.com/paper/2606.09266/) (2026)

## Abstract (arXiv)

Recently, Rendle has warned that the use of sampling-based top-k metrics might not suffice. This throws a number of recent studies on deep learning-based recommendation algorithms, and classic non-deep-learning algorithms using such a metric, into jeopardy. In this work, we thoroughly investigate the relationship between the sampling and global top-K Hit-Ratio (HR, or Recall), originally proposed by Koren[2] and extensively used by others. By formulating the problem of aligning sampling top-k (SHR@k) and global top-K (HR@K) Hit-Ratios through a mapping function f, so that SHR@k≈ HR@f(k), we demonstrate both theoretically and experimentally that the sampling top-k Hit-Ratio provides an accurate approximation of its global (exact) counterpart, and can consistently predict the correct winners (the same as indicate by their corresponding global Hit-Ratios).