mlsanityData to 14 Oct 2026

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

arXiv 2002.06305 ↗First posted 15 Feb 2020cs.CL cs.LGcited by 384

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.

Key facts. Cited by 384 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 67; 2022: 74; 2023: 68; 2024: 63; 2025: 54; 2026: 57.

384 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2022Editing Models with Task Arithmeticcited by 1,120
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  7. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2021True Few-Shot Learning with Language Modelscited by 344
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
38 more
  1. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  2. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  3. 2022Data Contamination: From Memorization to Exploitationcited by 236
  4. 2021How Many Data Points is a Prompt Worth?cited by 198
  5. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  6. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  7. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  8. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  9. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  10. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  11. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  12. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  13. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  14. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  15. 2022Efficient Few-Shot Learning Without Promptscited by 137
  16. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  17. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  18. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  19. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  20. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  21. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  22. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  23. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  24. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  25. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 92
  26. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  27. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 87
  28. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  29. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  30. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  31. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  32. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  33. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 69
  34. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  35. 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
  36. 2022Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AIcited by 58
  37. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  38. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53

50 most cited of 384 All 384 (JSON)