# L_DMI: An Information-theoretic Noise-robust Loss Function

arXiv: https://arxiv.org/abs/1909.03388 · HTML: https://mlsanity.com/paper/1909.03388/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2019.

## Built on by (most cited first, 26 of 26)

- [UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning](https://mlsanity.com/paper/2203.14542/) (2022)
- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels](https://mlsanity.com/paper/2102.05291/) (2021)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [Hard Sample Aware Noise Robust Learning for Histopathology Image Classification](https://mlsanity.com/paper/2112.03694/) (2021)
- [When Source-Free Domain Adaptation Meets Learning with Noisy Labels](https://mlsanity.com/paper/2301.13381/) (2023)
- [Contrastive Learning Improves Model Robustness Under Label Noise](https://mlsanity.com/paper/2104.08984/) (2021)
- [Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion](https://mlsanity.com/paper/2106.09291/) (2021)
- [Robust Long-Tailed Learning under Label Noise](https://mlsanity.com/paper/2108.11569/) (2021)
- [Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization](https://mlsanity.com/paper/2208.11351/) (2022)
- [Uncertainty Estimation via Response Scaling for Pseudo-mask Noise Mitigation in Weakly-supervised Semantic Segmentation](https://mlsanity.com/paper/2112.07431/) (2021)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation](https://mlsanity.com/paper/2207.13316/) (2022)
- [Fine-Grained Classification with Noisy Labels](https://mlsanity.com/paper/2303.02404/) (2023)
- [Which Samples Should be Learned First: Easy or Hard?](https://mlsanity.com/paper/2110.05481/) (2021)
- [Learning to Rectify for Robust Learning with Noisy Labels](https://mlsanity.com/paper/2111.04239/) (2021)
- [Sample Prior Guided Robust Model Learning to Suppress Noisy Labels](https://mlsanity.com/paper/2112.01197/) (2021)
- [Learning to Combat Noisy Labels via Classification Margins](https://mlsanity.com/paper/2102.00751/) (2021)
- [Truncate-Split-Contrast: A Framework for Learning from Mislabeled Videos](https://mlsanity.com/paper/2212.13495/) (2022)
- [Leveraging an Alignment Set in Tackling Instance-Dependent Label Noise](https://mlsanity.com/paper/2307.04868/) (2023)
- [Do We Really Need Gold Samples for Sample Weighting Under Label Noise?](https://mlsanity.com/paper/2104.09045/) (2021)
- [Learning from Long-Tailed Noisy Data with Sample Selection and Balanced Loss](https://mlsanity.com/paper/2211.10906/) (2022)
- [Combining Machine Learning Defenses without Conflicts](https://mlsanity.com/paper/2411.09776/) (2024)
- [An Ensemble Noise-Robust K-fold Cross-Validation Selection Method for Noisy Labels](https://mlsanity.com/paper/2107.02347/) (2021)
- [Program Synthesis Over Noisy Data with Guarantees](https://mlsanity.com/paper/2103.05030/) (2021)
- [Pairwise Similarity Distribution Clustering for Noisy Label Learning](https://mlsanity.com/paper/2404.01853/) (2024)

## Abstract (arXiv)

Accurately annotating large scale dataset is notoriously expensive both in time and in money. Although acquiring low-quality-annotated dataset can be much cheaper, it often badly damages the performance of trained models when using such dataset without particular treatment. Various methods have been proposed for learning with noisy labels. However, most methods only handle limited kinds of noise patterns, require auxiliary information or steps (e.g. , knowing or estimating the noise transition matrix), or lack theoretical justification. In this paper, we propose a novel information-theoretic loss function, LDMI, for training deep neural networks robust to label noise. The core of LDMI is a generalized version of mutual information, termed Determinant based Mutual Information (DMI), which is not only information-monotone but also relatively invariant. emphTo the best of our knowledge, LDMI is the first loss function that is provably robust to instance-independent label noise, regardless of noise pattern, and it can be applied to any existing classification neural networks straightforwardly without any auxiliary information. In addition to theoretical justification, we also empirically show that using LDMI outperforms all other counterparts in the classification task on both image dataset and natural language dataset include Fashion-MNIST, CIFAR-10, Dogs vs. Cats, MR with a variety of synthesized noise patterns and noise amounts, as well as a real-world dataset Clothing1M. Codes are available at https://github.com/Newbeeer/LDMI .