# Adaptive Label Smoothing

arXiv: https://arxiv.org/abs/2009.06432 · HTML: https://mlsanity.com/paper/2009.06432/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2020.

## Built on by (most cited first, 10 of 10)

- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [MAFA: Managing False Negatives for Vision-Language Pre-training](https://mlsanity.com/paper/2312.06112/) (2023)
- [Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation](https://mlsanity.com/paper/2210.13459/) (2022)
- [Diversifying Question Generation over Knowledge Base via External Natural Questions](https://mlsanity.com/paper/2309.14362/) (2023)
- [Adaptive Label Smoothing for Out-of-Distribution Detection](https://mlsanity.com/paper/2410.06134/) (2024)
- [Few-Shot Learning in Video and 3D Object Detection: A Survey](https://mlsanity.com/paper/2507.17079/) (2025)
- [Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validation](https://mlsanity.com/paper/2306.12189/) (2023)
- [Regularizing CNNs using Confusion Penalty Based Label Smoothing for Histopathology Images](https://mlsanity.com/paper/2403.10881/) (2024)
- [WHOSe Heritage: Classification of UNESCO World Heritage "Outstanding Universal Value" Documents with Soft Labels](https://mlsanity.com/paper/2104.05547/) (2021)
- [Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping](https://mlsanity.com/paper/2610.13909/) (2026)

## Abstract (arXiv)

This paper concerns the use of objectness measures to improve the calibration performance of Convolutional Neural Networks (CNNs). CNNs have proven to be very good classifiers and generally localize objects well; however, the loss functions typically used to train classification CNNs do not penalize inability to localize an object, nor do they take into account an object's relative size in the given image. During training on ImageNet-1K almost all approaches use random crops on the images and this transformation sometimes provides the CNN with background only samples. This causes the classifiers to depend on context. Context dependence is harmful for safety-critical applications. We present a novel approach to classification that combines the ideas of objectness and label smoothing during training. Unlike previous methods, we compute a smoothing factor that is adaptive based on relative object size within an image. This causes our approach to produce confidences that are grounded in the size of the object being classified instead of relying on context to make the correct predictions. We present extensive results using ImageNet to demonstrate that CNNs trained using adaptive label smoothing are much less likely to be overconfident in their predictions. We show qualitative results using class activation maps and quantitative results using classification and transfer learning tasks. Our approach is able to produce an order of magnitude reduction in confidence when predicting on context only images when compared to baselines. Using transfer learning, we gain 2.1mAP on MS COCO compared to the hard label approach.