# Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance

arXiv: https://arxiv.org/abs/2311.01108 · HTML: https://mlsanity.com/paper/2311.01108/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Learning with Instance-Dependent Label Noise: A Sample Sieve Approach](https://mlsanity.com/paper/2010.02347/) (2020)
- [Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning](https://mlsanity.com/paper/2006.07805/) (2020)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Augmentation Strategies for Learning with Noisy Labels](https://mlsanity.com/paper/2103.02130/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels](https://mlsanity.com/paper/2105.04522/) (2021)
- [Learning from Rules Generalizing Labeled Exemplars](https://mlsanity.com/paper/2004.06025/) (2020)

## Built on by (most cited first, 24 of 24)

- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Knowledge Graph-Enhanced Large Language Models via Path Selection](https://mlsanity.com/paper/2406.13862/) (2024)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models](https://mlsanity.com/paper/2407.02408/) (2024)
- [Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention](https://mlsanity.com/paper/2312.15033/) (2023)
- [Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance](https://mlsanity.com/paper/2504.02901/) (2025)
- [FastGAS: Fast Graph-based Annotation Selection for In-Context Learning](https://mlsanity.com/paper/2406.03730/) (2024)
- [Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort](https://mlsanity.com/paper/2407.08947/) (2024)
- [CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning](https://mlsanity.com/paper/2411.00873/) (2024)
- [Adversarial Training for Defense Against Label Poisoning Attacks](https://mlsanity.com/paper/2502.17121/) (2025)
- [Catching Chameleons: Detecting Evolving Disinformation Generated using Large Language Models](https://mlsanity.com/paper/2406.17992/) (2024)
- [Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement](https://mlsanity.com/paper/2505.19675/) (2025)
- [UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation](https://mlsanity.com/paper/2405.01022/) (2024)
- [Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?](https://mlsanity.com/paper/2407.00996/) (2024)
- [Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.03461/) (2024)
- [Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis](https://mlsanity.com/paper/2512.11912/) (2025)
- [Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models](https://mlsanity.com/paper/2504.03302/) (2025)
- [REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations](https://mlsanity.com/paper/2604.17289/) (2026)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [NaijaNLP: A Survey of Nigerian Low-Resource Languages](https://mlsanity.com/paper/2502.19784/) (2025)
- [Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning](https://mlsanity.com/paper/2510.10208/) (2025)
- [Sensitivity of Small Language Models to Fine-tuning Data Contamination](https://mlsanity.com/paper/2511.06763/) (2025)
- [Analyzing the Effect of Noise in LLM Fine-tuning](https://mlsanity.com/paper/2604.12469/) (2026)
- [Cross-Source Reasoning-based Correction for Author Name Disambiguation](https://mlsanity.com/paper/2606.08617/) (2026)

## Abstract (arXiv)

Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PLMs) have achieved substantial advancements in the field of natural language processing. However, in real-world scenarios, data labels are often noisy due to the complex annotation process, making it essential to develop strategies for fine-tuning PLMs with such noisy labels. To this end, we introduce an innovative approach for fine-tuning PLMs using noisy labels, which incorporates the guidance of Large Language Models (LLMs) like ChatGPT. This guidance assists in accurately distinguishing between clean and noisy samples and provides supplementary information beyond the noisy labels, thereby boosting the learning process during fine-tuning PLMs. Extensive experiments on synthetic and real-world noisy datasets further demonstrate the superior advantages of our framework over the state-of-the-art baselines.