mlsanityData to 7 Oct 2026

Analysing the Noise Model Error for Realistic Noisy Label Data

arXiv 2101.09763 ↗First posted 24 Jan 2021cs.LG cs.CL stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Distant and weak supervision allow to obtain large amounts of labeled training data quickly and cheaply, but these automatic annotations tend to contain a high amount of errors. A popular technique to overcome the negative effects of these noisy labels is noise modelling where the underlying noise process is modelled. In this work, we study the quality of these estimated noise models from the theoretical side by deriving the expected error of the noise model. Apart from evaluating the theoretical results on commonly used synthetic noise, we also publish NoisyNER, a new noisy label dataset from the NLP domain that was obtained through a realistic distant supervision technique. It provides seven sets of labels with differing noise patterns to evaluate different noise levels on the same instances. Parallel, clean labels are available making it possible to study scenarios where a small amount of gold-standard data can be leveraged. Our theoretical results and the corresponding experiments give insights into the factors that influence the noise model estimation like the noise distribution and the sampling technique.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 24 Jan 2021.

Builds on

30 of 54 references.

References · 30

  1. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  2. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  3. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  4. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  5. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  6. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  7. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  8. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  9. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  10. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  11. 2017CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noisecited by 181
  12. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
18 more
  1. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  2. 2017Learning From Noisy Large-Scale Datasets With Minimal Supervisioncited by 115
  3. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  4. 2013Classification with Asymmetric Label Noise: Consistency and Maximal Denoisingcited by 67
  5. 2017Learning with Bounded Instance- and Label-dependent Label Noisecited by 57
  6. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  7. 2020Named Entity Recognition without Labelled Data: A Weak Supervision Approachcited by 46
  8. 2020Part-dependent Label Noise: Towards Instance-dependent Label Noisecited by 40
  9. 2017Generalisation in Named Entity Recognition: A Quantitative Analysiscited by 25
  10. 2016Loss factorization, weakly supervised learning and label noise robustnesscited by 24
  11. 2017Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrixcited by 14
  12. 2019Towards Robust Learning with Different Label Noise Distributionscited by 14
  13. 2019Learning with Noisy Labels for Sentence-level Sentiment Classificationcited by 11
  14. 2016Learning when to trust distant supervision: An application to low-resource POS tagging using cross-lingual projectioncited by 8
  15. 2018Training a Neural Network in a Low-Resource Setting on Automatically Annotated Noisy Datacited by 8
  16. 2019Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labelscited by 5
  17. 2019Handling Noisy Labels for Robustly Learning from Self-Training Data for Low-Resource Sequence Labelingcited by 4
  18. 2019Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Frameworkcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 4; 2024: 4; 2025: 2.

14 papers.

Sort:

  1. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  2. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  3. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  4. 2022A law of adversarial risk, interpolation, and label noisecited by 8
  5. 2023NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processingcited by 8
  6. 2021Knodle: Modular Weakly Supervised Learning with PyTorchcited by 7
  7. 2023ConstraintMatch for Semi-constrained Clusteringcited by 2
  8. 2024Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?cited by 2
  9. 2024AlleNoise: large-scale text classification benchmark dataset with real-world label noisecited by 2
  10. 2024NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognitioncited by 1
  11. 2024NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classificationcited by 1
  12. 2023Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instancescited by 0
2 more
  1. 2025LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Trainingcited by 0
  2. 2025Sensitivity of Small Language Models to Fine-tuning Data Contaminationcited by 0