mlsanityData to 7 Oct 2026

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

arXiv 2211.01642 ↗First posted 3 Nov 2022cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation. Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models. In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse. We release our code at https://github.com/ZhangHaojie077/DPS

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 31 of its 59 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

31 of 59 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
19 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  9. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  10. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  11. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  12. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  13. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  14. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  15. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  16. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  17. 2019Where is the Information in a Deep Neural Network?cited by 40
  18. 2019On Adversarial Removal of Hypothesis-only Bias in Natural Language Inferencecited by 29
  19. 2018A Closer Look at Structured Pruning for Neural Network Compressioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 2; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  2. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  3. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  4. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  5. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  6. 2023E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translationcited by 5
  7. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  8. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  9. 2024Exploring Data-Efficient Adaptation of Large Language Models for Code Generationcited by 3
  10. 2024Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Textscited by 2
  11. 2024Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolationcited by 2
  12. 2024Exploring Selective Layer Fine-Tuning in Federated Learningcited by 2
6 more
  1. 2025Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Datacited by 1
  2. 2023Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimizationcited by 0
  3. 2024Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentationscited by 0
  4. 2025ModelTables: A Corpus of Tables about Modelscited by 0
  5. 2026Zero-shot adaptable task planning for autonomous construction robots: a comparative study of lightweight single and multi-AI agent systemscited by 0
  6. 2026Model soups need only one ingredientcited by 0