# Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification

arXiv: https://arxiv.org/abs/2204.09371 · HTML: https://mlsanity.com/paper/2204.09371/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey](https://mlsanity.com/paper/1912.05170/) (2019)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [An Effective Label Noise Model for DNN Text Classification](https://mlsanity.com/paper/1903.07507/) (2019)
- [Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels](https://mlsanity.com/paper/1910.06061/) (2019)

## Built on by (most cited first, 21 of 21)

- [Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance](https://mlsanity.com/paper/2311.01108/) (2023)
- [Detecting Label Errors by using Pre-Trained Language Models](https://mlsanity.com/paper/2205.12702/) (2022)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Exploring the Robustness of In-Context Learning with Noisy Labels](https://mlsanity.com/paper/2404.18191/) (2024)
- [animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics](https://mlsanity.com/paper/2406.01253/) (2024)
- [CrossSplit: Mitigating Label Noise Memorization through Data Splitting](https://mlsanity.com/paper/2212.01674/) (2022)
- [Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance](https://mlsanity.com/paper/2504.02901/) (2025)
- [Weaker Than You Think: A Critical Look at Weakly Supervised Learning](https://mlsanity.com/paper/2305.17442/) (2023)
- [Meta Self-Refinement for Robust Learning with Weak Supervision](https://mlsanity.com/paper/2205.07290/) (2022)
- [Adversarial Training for Defense Against Label Poisoning Attacks](https://mlsanity.com/paper/2502.17121/) (2025)
- [Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals](https://mlsanity.com/paper/2505.13972/) (2025)
- [Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation](https://mlsanity.com/paper/2506.03857/) (2025)
- [Handling Realistic Label Noise in BERT Text Classification](https://mlsanity.com/paper/2305.16337/) (2023)
- [Rumour detection using graph neural network and oversampling in benchmark Twitter dataset](https://mlsanity.com/paper/2212.10080/) (2022)
- [NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition](https://mlsanity.com/paper/2405.07609/) (2024)
- [NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification](https://mlsanity.com/paper/2407.06579/) (2024)
- [Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation](https://mlsanity.com/paper/2601.00263/) (2026)
- [iFlip: Iterative Feedback-driven Counterfactual Example Refinement](https://mlsanity.com/paper/2601.01446/) (2026)
- [How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detection](https://mlsanity.com/paper/2603.07346/) (2026)
- [Analyzing the Effect of Noise in LLM Fine-tuning](https://mlsanity.com/paper/2604.12469/) (2026)

## Abstract (arXiv)

Incorrect labels in training data occur when human annotators make mistakes or when the data is generated via weak or distant supervision. It has been shown that complex noise-handling techniques - by modeling, cleaning or filtering the noisy instances - are required to prevent models from fitting this label noise. However, we show in this work that, for text classification tasks with modern NLP models like BERT, over a variety of noise types, existing noisehandling methods do not always improve its performance, and may even deteriorate it, suggesting the need for further investigation. We also back our observations with a comprehensive analysis.