mlsanityData to 7 Oct 2026

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

arXiv 2202.12024 ↗First posted 24 Feb 2022cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Effectively finetuning pretrained language models (PLMs) is critical for their success in downstream tasks. However, PLMs may have risks in overfitting the pretraining tasks and data, which usually have gap with the target downstream tasks. Such gap may be difficult for existing PLM finetuning methods to overcome and lead to suboptimal performance. In this paper, we propose a very simple yet effective method named NoisyTune to help better finetune PLMs on downstream tasks by adding some noise to the parameters of PLMs before fine-tuning. More specifically, we propose a matrix-wise perturbing method which adds different uniform noises to different parameter matrices based on their standard deviations. In this way, the varied characteristics of different types of parameters in PLMs can be considered. Extensive experiments on both GLUE English benchmark and XTREME multilingual benchmark show NoisyTune can consistently empower the finetuning of different PLMs on different downstream tasks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

26 of 29 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
14 more
  1. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  2. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  3. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  4. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  5. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  6. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  7. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  8. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  9. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  10. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  11. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  12. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  13. 2021Empowering News Recommendation with Pre-trained Language Modelscited by 63
  14. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 15; 2025: 9; 2026: 3.

43 papers.

Sort:

By area:

  1. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  2. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  3. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 19
  4. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  5. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  6. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  7. 2023Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inferencecited by 13
  8. 2022HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbationcited by 11
  9. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  10. 2022Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizescited by 10
  11. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  12. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
31 more
  1. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  2. 2024Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Modelscited by 7
  3. 2024See the Unseen: Better Context-Consistent Knowledge-Editing by Noisescited by 5
  4. 2024Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learningcited by 5
  5. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4
  6. 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Modelscited by 4
  7. 2025Bi-Level Optimization for Self-Supervised AI-Generated Face Detectioncited by 4
  8. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  9. 2024VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrievalcited by 3
  10. 2025Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Datacited by 3
  11. 2025ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuningcited by 3
  12. 2022PATS: Sensitivity-aware Noisy Learning for Pretrained Language Modelscited by 2
  13. 2023Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fastcited by 2
  14. 2024Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolationcited by 2
  15. 2024Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?cited by 2
  16. 2024Tazza: Shuffling Neural Network Parameters for Secure and Private Federated Learningcited by 2
  17. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  18. 2023Improving Video Colorization by Test-Time Tuningcited by 1
  19. 2024Audio-Guided Fusion Techniques for Multimodal Emotion Analysiscited by 1
  20. 2024Exploring Accuracy-Fairness Trade-off in Large Language Modelscited by 1
  21. 2023Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimizationcited by 0
  22. 2023FastDoc: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomycited by 0
  23. 2023Controlled Randomness Improves the Performance of Transformer Modelscited by 0
  24. 2023Dynamic Corrective Self-Distillation for Better Fine-Tuning of Pretrained Modelscited by 0
  25. 2025ProtoBERT-LoRA: Parameter-Efficient Prototypical Finetuning for Immunotherapy Study Identificationcited by 0
  26. 2025Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injectioncited by 0
  27. 2025Enhanced Pre-training of Graph Neural Networks for Million-Scale Heterogeneous Graphscited by 0
  28. 2025Sensitivity of Small Language Models to Fine-tuning Data Contaminationcited by 0
  29. 2026Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classificationscited by 0
  30. 2026GRAIN: Group Aggregation via Min-Norm Objectivecited by 0
  31. 2026A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learningcited by 0