# Stable and Semi-stable Sampling Approaches for Continuously Used Samples

arXiv: https://arxiv.org/abs/2203.01381 · HTML: https://mlsanity.com/paper/2203.01381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 23 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 4 of 4)

- [Preserving Statistical Validity in Adaptive Data Analysis](https://mlsanity.com/paper/1411.2664/) (2014)
- [Effective Evaluation using Logged Bandit Feedback from Multiple Loggers](https://mlsanity.com/paper/1703.06180/) (2017)
- [Parallel Weighted Random Sampling](https://mlsanity.com/paper/1903.00227/) (2019)
- [Efficient Random Sampling -- Parallel, Vectorized, Cache-Efficient, and Online](https://mlsanity.com/paper/1610.05141/) (2016)

## Abstract (arXiv)

Information retrieval systems are usually measured by labeling the relevance of results corresponding to a sample of user queries. In practical search engines, such measurement needs to be performed continuously, such as daily or weekly. This creates a trade-off between (a) representativeness of query sample to current query traffic of the product; (b) labeling cost: if we keep the same query sample, results would be similar allowing us to reuse their labels; and (c) overfitting caused by continuous usage of same query sample. In this paper we explicitly formulate this tradeoff, propose two new variants -- Stable and Semi-stable -- to simple and weighted random sampling and show that they outperform existing approaches for the continuous usage settings, including monitoring/debugging search engine or comparing ranker candidates.