mlsanityData to 7 Oct 2026

Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification

arXiv 2204.09371 ↗First posted 20 Apr 2022cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Incorrect labels in training data occur when human annotators make mistakes or when the data is generated via weak or distant supervision. It has been shown that complex noise-handling techniques - by modeling, cleaning or filtering the noisy instances - are required to prevent models from fitting this label noise. However, we show in this work that, for text classification tasks with modern NLP models like BERT, over a variety of noise types, existing noisehandling methods do not always improve its performance, and may even deteriorate it, suggesting the need for further investigation. We also back our observations with a comprehensive analysis.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

16 of 23 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  5. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  6. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  7. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  8. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  9. 2020Does label smoothing mitigate label noise?cited by 204
  10. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
  11. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  12. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
4 more
  1. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  2. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  3. 2019An Effective Label Noise Model for DNN Text Classificationcited by 22
  4. 2019Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 5; 2025: 4; 2026: 4.

21 papers.

Sort:

  1. 2023Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidancecited by 24
  2. 2022Detecting Label Errors by using Pre-Trained Language Modelscited by 16
  3. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  4. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  5. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  6. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 13
  7. 2022CrossSplit: Mitigating Label Noise Memorization through Data Splittingcited by 12
  8. 2025Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistancecited by 11
  9. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  10. 2022Meta Self-Refinement for Robust Learning with Weak Supervisioncited by 7
  11. 2025Adversarial Training for Defense Against Label Poisoning Attackscited by 5
  12. 2025Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactualscited by 5
9 more
  1. 2025Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotationcited by 5
  2. 2023Handling Realistic Label Noise in BERT Text Classificationcited by 4
  3. 2022Rumour detection using graph neural network and oversampling in benchmark Twitter datasetcited by 1
  4. 2024NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognitioncited by 1
  5. 2024NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classificationcited by 1
  6. 2026Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generationcited by 1
  7. 2026iFlip: Iterative Feedback-driven Counterfactual Example Refinementcited by 1
  8. 2026How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detectioncited by 0
  9. 2026Analyzing the Effect of Noise in LLM Fine-tuningcited by 0