# Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms

arXiv: https://arxiv.org/abs/2509.24228 · HTML: https://mlsanity.com/paper/2509.24228/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Mixture Proportion Estimation via Kernel Embedding of Distributions](https://mlsanity.com/paper/1603.02501/) (2016)
- [Multiscale Positive-Unlabeled Detection of AI-Generated Texts](https://mlsanity.com/paper/2305.18149/) (2023)
- [On Symmetric Losses for Learning from Corrupted Labels](https://mlsanity.com/paper/1901.09314/) (2019)
- [Mixture Proportion Estimation and PU Learning: A Modern Approach](https://mlsanity.com/paper/2111.00980/) (2021)
- [Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training](https://mlsanity.com/paper/2006.11280/) (2020)
- [Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning](https://mlsanity.com/paper/1603.03130/) (2016)
- [Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective](https://mlsanity.com/paper/2212.02801/) (2022)
- [Beyond the Selected Completely At Random Assumption for Learning from Positive and Unlabeled Data](https://mlsanity.com/paper/1809.03207/) (2018)
- [Classification from Positive, Unlabeled and Biased Negative Data](https://mlsanity.com/paper/1810.00846/) (2018)
- [PU Learning for Matrix Completion](https://mlsanity.com/paper/1411.6081/) (2014)
- [Rethinking Class-Prior Estimation for Positive-Unlabeled Learning](https://mlsanity.com/paper/2002.03673/) (2020)
- [Realistic Evaluation of Deep Partial-Label Learning Algorithms](https://mlsanity.com/paper/2502.10184/) (2025)
- [Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends](https://mlsanity.com/paper/2310.04078/) (2023)
- [Mixture Proportion Estimation Beyond Irreducibility](https://mlsanity.com/paper/2306.01253/) (2023)
- [A Unified Approach to Count-Based Weakly-Supervised Learning](https://mlsanity.com/paper/2311.13718/) (2023)
- [A General Framework for Learning from Weak Supervision](https://mlsanity.com/paper/2402.01922/) (2024)
- [Robust Positive-Unlabeled Learning via Noise Negative Sample Self-correction](https://mlsanity.com/paper/2308.00279/) (2023)
- [Lambda: Learning Matchable Prior For Entity Alignment with Unlabeled Dangling Cases](https://mlsanity.com/paper/2403.10978/) (2024)
- [Unraveling the Impact of Heterophilic Structures on Graph Positive-Unlabeled Learning](https://mlsanity.com/paper/2405.19919/) (2024)
- [Positive-Unlabeled Diffusion Models for Preventing Sensitive Data Generation](https://mlsanity.com/paper/2503.03789/) (2025)

## Built on by (most cited first, 1 of 1)

- [Recent advances in weakly supervised learning: New supervision paradigms, assumption relaxations, and practical solutions](https://mlsanity.com/paper/2608.06896/) (2026)

## Abstract (arXiv)

Positive-unlabeled (PU) learning is a weakly supervised binary classification problem, in which the goal is to learn a binary classifier from only positive and unlabeled data, without access to negative data. In recent years, many PU learning algorithms have been developed to improve model performance. However, experimental settings are highly inconsistent, making it difficult to identify which algorithm performs better. In this paper, we propose the first PU learning benchmark to systematically compare PU learning algorithms. During our implementation, we identify subtle yet critical factors that affect the realistic and fair evaluation of PU learning algorithms. On the one hand, many PU learning algorithms rely on a validation set that includes negative data for model selection. This is unrealistic in traditional PU learning settings, where no negative data are available. To handle this problem, we systematically investigate model selection criteria for PU learning. On the other hand, PU learning involves different problem settings and corresponding solution families, i.e., the one-sample and two-sample settings. However, existing evaluation protocols are heavily biased towards the one-sample setting and neglect the significant difference between them. We identify the internal label shift problem of unlabeled training data for the one-sample setting and propose a simple yet effective calibration approach to ensure fair comparisons within and across families. We hope our framework will provide an accessible, realistic, and fair environment for evaluating PU learning algorithms in the future.