# Learning from Hard Labels with Additional Supervision on Non-Hard-Labeled Classes

arXiv: https://arxiv.org/abs/2507.18098 · HTML: https://mlsanity.com/paper/2507.18098/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Learning from Complementary Labels](https://mlsanity.com/paper/1705.07541/) (2017)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [Learning from Similarity-Confidence Data](https://mlsanity.com/paper/2102.06879/) (2021)
- [Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels](https://mlsanity.com/paper/2311.05265/) (2023)
- [Binary Classification with Confidence Difference](https://mlsanity.com/paper/2310.05632/) (2023)
- [Generalized Label Enhancement with Sample Correlations](https://mlsanity.com/paper/2004.03104/) (2020)
- [Label Information Bottleneck for Label Enhancement](https://mlsanity.com/paper/2303.06836/) (2023)

## Abstract (arXiv)

In scenarios where training data is limited due to observation costs or data scarcity, enriching the label information associated with each instance becomes crucial for building high-accuracy classification models. In such contexts, it is often feasible to obtain not only hard labels but also it additional supervision, such as the confidences for the hard labels. This setting naturally raises fundamental questions: it What kinds of additional supervision are intrinsically beneficial? And it how do they contribute to improved generalization performance? To address these questions, we propose a theoretical framework that treats both hard labels and additional supervision as probability distributions, and constructs soft labels through their affine combination. Our theoretical analysis reveals that the essential component of additional supervision is not the confidence score of the assigned hard label, but rather the information of the distribution over the non-hard-labeled classes. Moreover, we demonstrate that the additional supervision and the mixing coefficient contribute to the refinement of soft labels in complementary roles. Intuitively, in the probability simplex, the additional supervision determines the direction in which the deterministic distribution representing the hard label should be adjusted toward the true label distribution, while the mixing coefficient controls the step size along that direction. Through generalization error analysis, we theoretically characterize how the additional supervision and its mixing coefficient affect both the convergence rate and asymptotic value of the error bound. Finally, we experimentally demonstrate that, based on our theory, designing additional supervision can lead to improved classification accuracy, even when utilized in a simple manner.