mlsanityData to 14 Oct 2026

Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

arXiv 2109.05687 ↗First posted 13 Sep 2021cs.CL cs.AIcited by 146

Not on the map. Why?

Abstract (arXiv)

Recent pretrained language models extend from millions to billions of parameters. Thus the need to fine-tune an extremely large pretrained model with a limited training corpus arises in various downstream tasks. In this paper, we propose a straightforward yet effective fine-tuning technique, Child-Tuning, which updates a subset of parameters (called child network) of large pretrained models via strategically masking out the gradients of the non-child network during the backward process. Experiments on various downstream tasks in GLUE benchmark show that Child-Tuning consistently outperforms the vanilla fine-tuning by 1.5~8.6 average score among four different pretrained models, and surpasses the prior fine-tuning techniques by 0.6~1.3 points. Furthermore, empirical results on domain transfer and task transfer show that Child-Tuning can obtain better generalization performance by large margins.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

31 of 45 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
19 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  3. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  4. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  5. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  6. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  8. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  9. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  10. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  11. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  12. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  13. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  14. 2020Dynamic Model Pruning with Feedbackcited by 125
  15. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  16. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 48
  17. 2017Natural Language Inference over Interaction Spacecited by 34
  18. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26
  19. 2016Reducing the Model Order of Deep Neural Networks Using Information Theorycited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 26; 2023: 41; 2024: 42; 2025: 29; 2026: 6.

146 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  3. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  6. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  7. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  8. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  9. 2022Neural Prompt Searchcited by 135
  10. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  11. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  12. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 84
38 more
  1. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  2. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  3. 2022Transferability in Deep Learning: A Surveycited by 57
  4. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  5. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  6. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  7. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  8. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  9. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  10. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  11. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  12. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  13. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  14. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  15. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  16. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  17. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  18. 2024ResAdapter: Domain Consistent Resolution Adapter for Diffusion Modelscited by 23
  19. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  20. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  21. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21
  22. 2023SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channelscited by 20
  23. 2023Weakly Supervised Detection of Hallucinations in LLM Activationscited by 20
  24. 2022Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworkscited by 18
  25. 2022Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptivelycited by 18
  26. 2023One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningcited by 18
  27. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17
  28. 2024Sparse High Rank Adapterscited by 17
  29. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 16
  30. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 16
  31. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  32. 2025FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentationcited by 16
  33. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  34. 2022Parameter-Efficient Tuning Makes a Good Classification Headcited by 15
  35. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  36. 2023Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inferencecited by 13
  37. 2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
  38. 2024Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspectivecited by 12

50 most cited of 146 All 146 (JSON)