mlsanityData to 7 Oct 2026

Detecting Label Errors by using Pre-Trained Language Models

arXiv 2205.12702 ↗First posted 25 May 2022cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We show that large pre-trained language models are inherently highly capable of identifying label errors in natural language datasets: simply examining out-of-sample data points in descending order of fine-tuned task loss significantly outperforms more complex error-detection mechanisms proposed in previous work. To this end, we contribute a novel method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. We show that this noise has similar properties to real, hand-verified label errors, and is harder to detect than existing synthetic noise, creating challenges for model robustness. We argue that human-originated noise is a better standard for evaluation than synthetic noise. Finally, we use crowdsourced verification to evaluate the detection of real errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than existing models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

43 of 79 references.

References · 43

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  10. 2015Image-based Recommendations on Styles and Substitutescited by 702
  11. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  12. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
31 more
  1. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  3. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  4. 2020Calibration of Pre-trained Transformerscited by 307
  5. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  6. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  7. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  9. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  10. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  11. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  12. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  13. 2021Assessing Generalization of SGD via Disagreementcited by 116
  14. 2020Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noisecited by 82
  15. 2021FINE Samples for Learning with Noisy Labelscited by 82
  16. 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
  17. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  18. 2021Understanding Hard Negatives in Noise Contrastive Estimationcited by 39
  19. 2021Learning from Noisy Labels for Entity-Centric Information Extractioncited by 34
  20. 2020Label Noise Types and Their Effects on Deep Learningcited by 29
  21. 2019Identifying Mislabeled Instances in Classification Datasetscited by 26
  22. 2022Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classificationcited by 21
  23. 2022Symphony: Composing Interactive Interfaces for Machine Learningcited by 18
  24. 2021Learning From Long-Tailed Data With Noisy Labelscited by 17
  25. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  26. 2015Auxiliary Image Regularization for Deep CNNs with Noisy Labelscited by 14
  27. 2021Constrained Instance and Class Reweighting for Robust Learning under Label Noisecited by 12
  28. 2021The Hidden Cost of Using Amazon Mechanical Turk for Researchcited by 6
  29. 2021DataCLUE: A Benchmark Suite for Data-centric NLPcited by 6
  30. 2021Multiplicative Reweighting for Robust Neural Network Optimizationcited by 4
  31. 2021Rethinking Noisy Label Models: Labeler-Dependent Noise with Adversarial Awarenesscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  2. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 40
  3. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  4. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  5. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  6. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  7. 2023Neural Relation Graph: A Unified Framework for Identifying Label Noise and Outlier Datacited by 20
  8. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  9. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  10. 2023ActiveAED: A Human in the Loop Improves Annotation Error Detectioncited by 9
  11. 2023On Evaluation of Document Classification using RVL-CDIPcited by 5
  12. 2023Handling Realistic Label Noise in BERT Text Classificationcited by 4
4 more
  1. 2024The Influence of Faulty Labels in Data Sets on Human Pose Estimationcited by 2
  2. 2024Uncovering Misattributed Suicide Causes through Annotation Inconsistency Detection in Death Investigation Notescited by 1
  3. 2024Robust Testing for Deep Learning using Human Label Noisecited by 0
  4. 2026Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agentscited by 0