# Adaptive Regularization of Labels

arXiv: https://arxiv.org/abs/1908.05474 · HTML: https://mlsanity.com/paper/1908.05474/

## Key facts

- Cited by 15 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Aug 2019.

## Built on by (most cited first, 15 of 15)

- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [Robust Cross-Modal Representation Learning with Progressive Self-Distillation](https://mlsanity.com/paper/2204.04588/) (2022)
- [Knowledge Distillation on Graphs: A Survey](https://mlsanity.com/paper/2302.00219/) (2023)
- [Pre-training Distillation for Large Language Models: A Design Space Exploration](https://mlsanity.com/paper/2410.16215/) (2024)
- [Function-Consistent Feature Distillation](https://mlsanity.com/paper/2304.11832/) (2023)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [Adaptive Label Smoothing To Regularize Large-Scale Graph Training](https://mlsanity.com/paper/2108.13555/) (2021)
- [Knowledge Distillation of Transformer-based Language Models Revisited](https://mlsanity.com/paper/2206.14366/) (2022)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [Smooth and Stepwise Self-Distillation for Object Detection](https://mlsanity.com/paper/2303.05015/) (2023)
- [Controlling the Quality of Distillation in Response-Based Network Compression](https://mlsanity.com/paper/2112.10047/) (2021)
- [Triplet Knowledge Distillation](https://mlsanity.com/paper/2305.15975/) (2023)
- [CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation](https://mlsanity.com/paper/2511.21503/) (2025)

## Abstract (arXiv)

Recently, a variety of regularization techniques have been widely applied in deep neural networks, such as dropout, batch normalization, data augmentation, and so on. These methods mainly focus on the regularization of weight parameters to prevent overfitting effectively. In addition, label regularization techniques such as label smoothing and label disturbance have also been proposed with the motivation of adding a stochastic perturbation to labels. In this paper, we propose a novel adaptive label regularization method, which enables the neural network to learn from the erroneous experience and update the optimal label representation online. On the other hand, compared with knowledge distillation, which learns the correlation of categories using teacher network, our proposed method requires only a minuscule increase in parameters without cumbersome teacher network. Furthermore, we evaluate our method on CIFAR-10/CIFAR-100/ImageNet datasets for image recognition tasks and AGNews/Yahoo/Yelp-Full datasets for text classification tasks. The empirical results show significant improvement under all experimental settings.