mlsanityData to 14 Oct 2026

Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance

arXiv 2311.01108 ↗First posted 2 Nov 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PLMs) have achieved substantial advancements in the field of natural language processing. However, in real-world scenarios, data labels are often noisy due to the complex annotation process, making it essential to develop strategies for fine-tuning PLMs with such noisy labels. To this end, we introduce an innovative approach for fine-tuning PLMs using noisy labels, which incorporates the guidance of Large Language Models (LLMs) like ChatGPT. This guidance assists in accurately distinguishing between clean and noisy samples and provides supplementary information beyond the noisy labels, thereby boosting the learning process during fine-tuning PLMs. Extensive experiments on synthetic and real-world noisy datasets further demonstrate the superior advantages of our framework over the state-of-the-art baselines.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

39 of 54 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  7. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  8. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  9. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  10. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  11. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  12. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
27 more
  1. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  2. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  3. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  5. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  6. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  7. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  8. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  9. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  10. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  11. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  12. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  13. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
  14. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  15. 2021Augmentation Strategies for Learning with Noisy Labelscited by 86
  16. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  17. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  18. 2020Learning from Rules Generalizing Labeled Exemplarscited by 59
  19. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  20. 2021Instance-dependent Label-noise Learning under a Structural Causal Modelcited by 50
  21. 2021Learning from Noisy Labels for Entity-Centric Information Extractioncited by 34
  22. 2019NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extractioncited by 24
  23. 2022From Noisy Prediction to True Label: Noisy Prediction Calibration via Generative Modelcited by 22
  24. 2022Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classificationcited by 21
  25. 2022Centrality and Consistency: Two-Stage Clean Samples Identification for Learning with Instance-Dependent Noisy Labelscited by 16
  26. 2021Hierarchical Heterogeneous Graph Representation Learning for Short Text Classificationcited by 13
  27. 2023Fair Few-shot Learning with Auxiliary Setscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 8; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networkscited by 56
  2. 2024Knowledge Graph-Enhanced Large Language Models via Path Selectioncited by 28
  3. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
  4. 2024CEB: Compositional Evaluation Benchmark for Fairness in Large Language Modelscited by 18
  5. 2023Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Interventioncited by 12
  6. 2025Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistancecited by 11
  7. 2024FastGAS: Fast Graph-based Annotation Selection for In-Context Learningcited by 8
  8. 2024Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effortcited by 7
  9. 2024CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learningcited by 6
  10. 2025Adversarial Training for Defense Against Label Poisoning Attackscited by 6
  11. 2024Catching Chameleons: Detecting Evolving Disinformation Generated using Large Language Modelscited by 5
  12. 2025Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinementcited by 4
12 more
  1. 2024UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generationcited by 3
  2. 2024Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?cited by 2
  3. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  4. 2025Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysiscited by 2
  5. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  6. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  7. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  8. 2025NaijaNLP: A Survey of Nigerian Low-Resource Languagescited by 0
  9. 2025Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learningcited by 0
  10. 2025Sensitivity of Small Language Models to Fine-tuning Data Contaminationcited by 0
  11. 2026Analyzing the Effect of Noise in LLM Fine-tuningcited by 0
  12. 2026Cross-Source Reasoning-based Correction for Author Name Disambiguationcited by 0