# A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks

arXiv: https://arxiv.org/abs/2412.09579 · HTML: https://mlsanity.com/paper/2412.09579/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 1 of its 24 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 1 of 1)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)

## Built on by (most cited first, 5 of 5)

- [LGAI-EMBEDDING-Preview Technical Report](https://mlsanity.com/paper/2506.07438/) (2025)
- [The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model](https://mlsanity.com/paper/2501.16226/) (2025)
- [Distributions In, Distributions Out: The Case for Soft-Label Training](https://mlsanity.com/paper/2511.14117/) (2025)
- [Task-Specific Knowledge Distillation via Intermediate Probes](https://mlsanity.com/paper/2603.12270/) (2026)
- [AI of the People, by the People, for the People: A Social Choice Approach to Collective Control of Artificial Intelligence](https://mlsanity.com/paper/2605.16291/) (2026)

## Abstract (arXiv)

Knowledge distillation, where a small student model learns from a pre-trained large teacher model, has achieved substantial empirical success since the seminal work of citephinton2015distilling. Despite prior theoretical studies exploring the benefits of knowledge distillation, an important question remains unanswered: why does soft-label training from the teacher require significantly fewer neurons than directly training a small neural network with hard labels? To address this, we first present motivating experimental results using simple neural network models on a binary classification problem. These results demonstrate that soft-label training consistently outperforms hard-label training in accuracy, with the performance gap becoming more pronounced as the dataset becomes increasingly difficult to classify. We then substantiate these observations with a theoretical contribution based on two-layer neural network models. Specifically, we show that soft-label training using gradient descent requires only Oleft(frac1γ2 εright) neurons to achieve a classification loss averaged over epochs smaller than some ε > 0, where γ is the separation margin of the limiting kernel. In contrast, hard-label training requires Oleft(frac1γ4 · lnleft(frac1εright)right) neurons, as derived from an adapted version of the gradient descent analysis in citepji2020polylogarithmic. This implies that when γ ≤ ε, i.e., when the dataset is challenging to classify, the neuron requirement for soft-label training can be significantly lower than that for hard-label training. Finally, we present experimental results on deep neural networks, further validating these theoretical findings.