# CleaR: Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning

arXiv: https://arxiv.org/abs/2411.00873 · HTML: https://mlsanity.com/paper/2411.00873/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning](https://mlsanity.com/paper/2210.17451/) (2022)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Revisiting Parameter-Efficient Tuning: Are We Really There Yet?](https://mlsanity.com/paper/2202.07962/) (2022)
- [Parameter-Efficient Fine-Tuning Design Spaces](https://mlsanity.com/paper/2301.01821/) (2023)
- [Parameter-Efficient Fine-Tuning without Introducing New Latency](https://mlsanity.com/paper/2305.16742/) (2023)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training](https://mlsanity.com/paper/2109.05003/) (2021)
- [Learning from Noisy Labels for Entity-Centric Information Extraction](https://mlsanity.com/paper/2104.08656/) (2021)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models](https://mlsanity.com/paper/2305.19249/) (2023)
- [Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation](https://mlsanity.com/paper/2002.10345/) (2020)
- [Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance](https://mlsanity.com/paper/2311.01108/) (2023)
- [An Effective Label Noise Model for DNN Text Classification](https://mlsanity.com/paper/1903.07507/) (2019)
- [Distant Learning for Entity Linking with Automatic Noise Detection](https://mlsanity.com/paper/1905.07189/) (2019)
- [DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling](https://mlsanity.com/paper/2305.19395/) (2023)

## Built on by (most cited first, 6 of 6)

- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [Efficient Learning for Product Attributes with Compact Multimodal Models](https://mlsanity.com/paper/2507.19679/) (2025)
- [Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning](https://mlsanity.com/paper/2510.10208/) (2025)
- [Analyzing the Effect of Noise in LLM Fine-tuning](https://mlsanity.com/paper/2604.12469/) (2026)
- [Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2606.10610/) (2026)
- [LoRA's Second Descent Extends Beyond Parameter Parity](https://mlsanity.com/paper/2610.04507/) (2026)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) has enabled the efficient optimization of cumbersome language models in real-world settings. However, as datasets in such environments often contain noisy labels that adversely affect performance, PEFT methods are inevitably exposed to noisy labels. Despite this challenge, the adaptability of PEFT to noisy environments remains underexplored. To bridge this gap, we investigate various PEFT methods under noisy labels. Interestingly, our findings reveal that PEFT has difficulty in memorizing noisy labels due to its inherently limited capacity, resulting in robustness. However, we also find that such limited capacity simultaneously makes PEFT more vulnerable to interference of noisy labels, impeding the learning of clean samples. To address this issue, we propose Clean Routing (CleaR), a novel routing-based PEFT approach that adaptively activates PEFT modules. In CleaR, PEFT modules are preferentially exposed to clean data while bypassing the noisy ones, thereby minimizing the noisy influence. To verify the efficacy of CleaR, we perform extensive experiments on diverse configurations of noisy labels. The results convincingly demonstrate that CleaR leads to substantially improved performance in noisy environments.