# Analysing the Noise Model Error for Realistic Noisy Label Data

arXiv: https://arxiv.org/abs/2101.09763 · HTML: https://mlsanity.com/paper/2101.09763/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 24 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey](https://mlsanity.com/paper/1912.05170/) (2019)
- [Classification with Asymmetric Label Noise: Consistency and Maximal Denoising](https://mlsanity.com/paper/1303.1208/) (2013)
- [Learning with Bounded Instance- and Label-dependent Label Noise](https://mlsanity.com/paper/1709.03768/) (2017)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [Part-dependent Label Noise: Towards Instance-dependent Label Noise](https://mlsanity.com/paper/2006.07836/) (2020)
- [Generalisation in Named Entity Recognition: A Quantitative Analysis](https://mlsanity.com/paper/1701.02877/) (2017)
- [Loss factorization, weakly supervised learning and label noise robustness](https://mlsanity.com/paper/1602.02450/) (2016)
- [Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix](https://mlsanity.com/paper/1705.03995/) (2017)
- [Towards Robust Learning with Different Label Noise Distributions](https://mlsanity.com/paper/1912.08741/) (2019)
- [Learning with Noisy Labels for Sentence-level Sentiment Classification](https://mlsanity.com/paper/1909.00124/) (2019)
- [Learning when to trust distant supervision: An application to low-resource POS tagging using cross-lingual projection](https://mlsanity.com/paper/1607.01133/) (2016)
- [Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Data](https://mlsanity.com/paper/1807.00745/) (2018)
- [Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels](https://mlsanity.com/paper/1910.06061/) (2019)
- [Handling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labeling](https://mlsanity.com/paper/1903.12008/) (2019)
- [Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Framework](https://mlsanity.com/paper/1909.05448/) (2019)

## Built on by (most cited first, 14 of 14)

- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [A law of adversarial risk, interpolation, and label noise](https://mlsanity.com/paper/2207.03933/) (2022)
- [NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing](https://mlsanity.com/paper/2305.10709/) (2023)
- [Knodle: Modular Weakly Supervised Learning with PyTorch](https://mlsanity.com/paper/2104.11557/) (2021)
- [ConstraintMatch for Semi-constrained Clustering](https://mlsanity.com/paper/2311.15395/) (2023)
- [Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?](https://mlsanity.com/paper/2407.00996/) (2024)
- [AlleNoise: large-scale text classification benchmark dataset with real-world label noise](https://mlsanity.com/paper/2407.10992/) (2024)
- [NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition](https://mlsanity.com/paper/2405.07609/) (2024)
- [NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification](https://mlsanity.com/paper/2407.06579/) (2024)
- [Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances](https://mlsanity.com/paper/2310.16790/) (2023)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Sensitivity of Small Language Models to Fine-tuning Data Contamination](https://mlsanity.com/paper/2511.06763/) (2025)

## Abstract (arXiv)

Distant and weak supervision allow to obtain large amounts of labeled training data quickly and cheaply, but these automatic annotations tend to contain a high amount of errors. A popular technique to overcome the negative effects of these noisy labels is noise modelling where the underlying noise process is modelled. In this work, we study the quality of these estimated noise models from the theoretical side by deriving the expected error of the noise model. Apart from evaluating the theoretical results on commonly used synthetic noise, we also publish NoisyNER, a new noisy label dataset from the NLP domain that was obtained through a realistic distant supervision technique. It provides seven sets of labels with differing noise patterns to evaluate different noise levels on the same instances. Parallel, clean labels are available making it possible to study scenarios where a small amount of gold-standard data can be leveraged. Our theoretical results and the corresponding experiments give insights into the factors that influence the noise model estimation like the noise distribution and the sampling technique.