# Realistic Evaluation of Deep Partial-Label Learning Algorithms

arXiv: https://arxiv.org/abs/2502.10184 · HTML: https://mlsanity.com/paper/2502.10184/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 81 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Progressive Identification of True Labels for Partial-Label Learning](https://mlsanity.com/paper/2002.08053/) (2020)
- [Learning with Biased Complementary Labels](https://mlsanity.com/paper/1711.09535/) (2017)
- [Provably Consistent Partial-Label Learning](https://mlsanity.com/paper/2007.08929/) (2020)
- [Learning from Complementary Labels](https://mlsanity.com/paper/1705.07541/) (2017)
- [Leveraged Weighted Loss for Partial Label Learning](https://mlsanity.com/paper/2106.05731/) (2021)
- [Complementary-Label Learning for Arbitrary Losses and Models](https://mlsanity.com/paper/1810.04327/) (2018)
- [Learning with Multiple Complementary Labels](https://mlsanity.com/paper/1912.12927/) (2019)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)
- [Instance-Dependent Partial Label Learning](https://mlsanity.com/paper/2110.12911/) (2021)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [Learning from Ambiguously Labeled Face Images](https://mlsanity.com/paper/1702.04455/) (2017)
- [Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels](https://mlsanity.com/paper/2007.02235/) (2020)
- [GM-PLL: Graph Matching based Partial Label Learning](https://mlsanity.com/paper/1901.03073/) (2019)
- [Progressive Purification for Instance-Dependent Partial Label Learning](https://mlsanity.com/paper/2206.00830/) (2022)
- [Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding](https://mlsanity.com/paper/1602.05307/) (2016)
- [A Regularization Approach for Instance-Based Superset Label Learning](https://mlsanity.com/paper/1904.02832/) (2019)
- [On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective](https://mlsanity.com/paper/2011.09172/) (2020)
- [PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning](https://mlsanity.com/paper/2201.08984/) (2022)
- [On the Robustness of Average Losses for Partial-Label Learning](https://mlsanity.com/paper/2106.06152/) (2021)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning](https://mlsanity.com/paper/2303.10365/) (2023)

## Built on by (most cited first, 12 of 12)

- [Rethinking Consistent Multi-Label Classification Under Inexact Supervision](https://mlsanity.com/paper/2510.04091/) (2025)
- [Tuning the Right Foundation Models is What you Need for Partial Label Learning](https://mlsanity.com/paper/2506.05027/) (2025)
- [Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms](https://mlsanity.com/paper/2509.24228/) (2025)
- [Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning](https://mlsanity.com/paper/2512.17788/) (2025)
- [crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels](https://mlsanity.com/paper/2504.09085/) (2025)
- [CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels](https://mlsanity.com/paper/2507.11807/) (2025)
- [Learning Robust Diffusion Models from Imprecise Supervision](https://mlsanity.com/paper/2510.03016/) (2025)
- [Progressive Power Homotopy for Non-convex Optimization](https://mlsanity.com/paper/2601.15915/) (2026)
- [Mitigating Instance Entanglement in Instance-Dependent Partial Label Learning](https://mlsanity.com/paper/2603.04825/) (2026)
- [Risk-Consistent Multiclass Learning from Random Label-Subset Membership Queries](https://mlsanity.com/paper/2605.07413/) (2026)
- [Recent advances in weakly supervised learning: New supervision paradigms, assumption relaxations, and practical solutions](https://mlsanity.com/paper/2608.06896/) (2026)
- [Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning](https://mlsanity.com/paper/2610.06234/) (2026)

## Abstract (arXiv)

Partial-label learning (PLL) is a weakly supervised learning problem in which each example is associated with multiple candidate labels and only one is the true label. In recent years, many deep PLL algorithms have been developed to improve model performance. However, we find that some early developed algorithms are often underestimated and can outperform many later algorithms with complicated designs. In this paper, we delve into the empirical perspective of PLL and identify several critical but previously overlooked issues. First, model selection for PLL is non-trivial, but has never been systematically studied. Second, the experimental settings are highly inconsistent, making it difficult to evaluate the effectiveness of the algorithms. Third, there is a lack of real-world image datasets that can be compatible with modern network architectures. Based on these findings, we propose PLENCH, the first Partial-Label learning bENCHmark to systematically compare state-of-the-art deep PLL algorithms. We investigate the model selection problem for PLL for the first time, and propose novel model selection criteria with theoretical guarantees. We also create Partial-Label CIFAR-10 (PLCIFAR10), an image dataset of human-annotated partial labels collected from Amazon Mechanical Turk, to provide a testbed for evaluating the performance of PLL algorithms in more realistic scenarios. Researchers can quickly and conveniently perform a comprehensive and fair evaluation and verify the effectiveness of newly developed algorithms based on PLENCH. We hope that PLENCH will facilitate standardized, fair, and practical evaluation of PLL algorithms in the future.