# Handling Realistic Label Noise in BERT Text Classification

arXiv: https://arxiv.org/abs/2305.16337 · HTML: https://mlsanity.com/paper/2305.16337/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [Learning from Noisy Labels with Distillation](https://mlsanity.com/paper/1703.02391/) (2017)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Label Noise Types and Their Effects on Deep Learning](https://mlsanity.com/paper/2003.10471/) (2020)
- [Learning with Different Amounts of Annotation: From Zero to Many Labels](https://mlsanity.com/paper/2109.04408/) (2021)
- [DSGAN: Generative Adversarial Training for Distant Supervision Relation Extraction](https://mlsanity.com/paper/1805.09929/) (2018)
- [An Effective Label Noise Model for DNN Text Classification](https://mlsanity.com/paper/1903.07507/) (2019)
- [Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification](https://mlsanity.com/paper/2204.09371/) (2022)
- [Detecting Label Errors by using Pre-Trained Language Models](https://mlsanity.com/paper/2205.12702/) (2022)
- [SENT: Sentence-level Distant Relation Extraction via Negative Training](https://mlsanity.com/paper/2106.11566/) (2021)

## Built on by (most cited first, 4 of 4)

- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation](https://mlsanity.com/paper/2405.01022/) (2024)
- [ELM: A Hybrid Ensemble of Language Models for Automated Tumor Group Classification in Population-Based Cancer Registries](https://mlsanity.com/paper/2503.21800/) (2025)
- [Bridging AI Innovation and Healthcare Needs: Lessons Learned from Incorporating Modern NLP at The BC Cancer Registry](https://mlsanity.com/paper/2508.09991/) (2025)

## Abstract (arXiv)

Labels noise refers to errors in training labels caused by cheap data annotation methods, such as web scraping or crowd-sourcing, which can be detrimental to the performance of supervised classifiers. Several methods have been proposed to counteract the effect of random label noise in supervised classification, and some studies have shown that BERT is already robust against high rates of randomly injected label noise. However, real label noise is not random; rather, it is often correlated with input features or other annotator-specific factors. In this paper, we evaluate BERT in the presence of two types of realistic label noise: feature-dependent label noise, and synthetic label noise from annotator disagreements. We show that the presence of these types of noise significantly degrades BERT classification performance. To improve robustness, we evaluate different types of ensembles and noise-cleaning methods and compare their effectiveness against label noise across different datasets.