# On the Reliability of Sampling Strategies in Offline Recommender Evaluation

arXiv: https://arxiv.org/abs/2508.05398 · HTML: https://mlsanity.com/paper/2508.05398/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems](https://mlsanity.com/paper/2202.10842/) (2022)
- [Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback](https://mlsanity.com/paper/1909.03601/) (2019)
- [KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos](https://mlsanity.com/paper/2208.08696/) (2022)
- [The Unfairness of Popularity Bias in Recommendation](https://mlsanity.com/paper/1907.13286/) (2019)
- [Modeling User Exposure in Recommendation](https://mlsanity.com/paper/1510.07025/) (2015)
- [Metadata Embeddings for User and Item Cold-start Recommendations](https://mlsanity.com/paper/1507.08439/) (2015)
- [A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models](https://mlsanity.com/paper/2107.13045/) (2021)
- [On Sampling Top-K Recommendation Evaluation](https://mlsanity.com/paper/2106.10621/) (2021)
- [Correcting Exposure Bias for Link Recommendation](https://mlsanity.com/paper/2106.07041/) (2021)
- [Fairness of Exposure in Light of Incomplete Exposure Estimation](https://mlsanity.com/paper/2205.12901/) (2022)
- [ContentWise Impressions: An Industrial Dataset with Impressions Included](https://mlsanity.com/paper/2008.01212/) (2020)
- [Debiased Recommendation with Noisy Feedback](https://mlsanity.com/paper/2406.17182/) (2024)
- [Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems](https://mlsanity.com/paper/2404.18640/) (2024)
- [Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sampling](https://mlsanity.com/paper/2104.15046/) (2021)
- [Candidate Set Sampling for Evaluating Top-N Recommendation](https://mlsanity.com/paper/2309.11723/) (2023)

## Abstract (arXiv)

Offline evaluation plays a central role in benchmarking recommender systems when online testing is impractical or risky. However, it is susceptible to two key sources of bias: exposure bias, where users only interact with items they are shown, and sampling bias, introduced when evaluation is performed on a subset of logged items rather than the full catalog. While prior work has proposed methods to mitigate sampling bias, these are typically assessed on fixed logged datasets rather than for their ability to support reliable model comparisons under varying exposure conditions or relative to true user preferences. In this paper, we investigate how different combinations of logging and sampling choices affect the reliability of offline evaluation. Using a fully observed dataset as ground truth, we systematically simulate diverse exposure biases and assess the reliability of common sampling strategies along four dimensions: sampling resolution (recommender model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth). Our findings highlight when and how sampling distorts evaluation outcomes and offer practical guidance for selecting strategies that yield faithful and robust offline comparisons.