# Active learning with biased non-response to label requests

arXiv: https://arxiv.org/abs/2312.08150 · HTML: https://mlsanity.com/paper/2312.08150/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 43 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://mlsanity.com/paper/1804.07931/) (2018)
- [On Statistical Bias In Active Learning: How and When To Fix It](https://mlsanity.com/paper/2101.11665/) (2021)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [Online Learning with Abstention](https://mlsanity.com/paper/1703.03478/) (2017)
- [Tree based Progressive Regression Model for Watch-Time Prediction in Short-video Recommendation](https://mlsanity.com/paper/2306.03392/) (2023)
- [Streaming Active Learning Strategies for Real-Life Credit Card Fraud Detection: Assessment and Visualization](https://mlsanity.com/paper/1804.07481/) (2018)
- [Active Learning from Imperfect Labelers](https://mlsanity.com/paper/1610.09730/) (2016)
- [Machine Learning for Fraud Detection in E-Commerce: A Research Agenda](https://mlsanity.com/paper/2107.01979/) (2021)
- [Margin-based sampling in high dimensions: When being active is less efficient than staying passive](https://mlsanity.com/paper/2212.00772/) (2022)
- [Bayesian Active Learning With Abstention Feedbacks](https://mlsanity.com/paper/1906.02179/) (2019)

## Abstract (arXiv)

Active learning can improve the efficiency of training prediction models by identifying the most informative new labels to acquire. However, non-response to label requests can impact active learning's effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data, demonstrating that biased non-response is particularly detrimental to model performance. We argue that biased non-response is likely in contexts where the labelling process, by nature, relies on user interactions. To mitigate the impact of biased non-response, we propose a cost-based correction to the sampling strategy--the Upper Confidence Bound of the Expected Utility (UCB-EU)--that can, plausibly, be applied to any active learning algorithm. Through experiments, we demonstrate that our method successfully reduces the harm from labelling non-response in many settings. However, we also characterise settings where the non-response bias in the annotations remains detrimental under UCB-EU for specific sampling methods and data generating processes. Finally, we evaluate our method on a real-world dataset from an e-commerce platform. We show that UCB-EU yields substantial performance improvements to conversion models that are trained on clicked impressions. Most generally, this research serves to both better conceptualise the interplay between types of non-response and model improvements via active learning, and to provide a practical, easy-to-implement correction that mitigates model degradation.