mlsanityData to 7 Oct 2026

Measuring the Instability of Fine-Tuning

arXiv 2302.07778 ↗First posted 15 Feb 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets. Many previous studies have investigated this instability and proposed methods to mitigate it. However, most studies only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. In this paper, we analyze SD and six other measures quantifying instability at different levels of granularity. Moreover, we propose a systematic framework to evaluate the validity of these measures. Finally, we analyze the consistency and difference between different measures by reassessing existing instability mitigation methods. We hope our results will inform the development of better measurements of fine-tuning instability.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

32 of 49 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
20 more
  1. 2018Neural Network Acceptability Judgmentscited by 952
  2. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  3. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  4. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  5. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  6. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  7. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  8. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  9. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  10. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  11. 2020On Linear Identifiability of Learned Representationscited by 103
  12. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  13. 2021Similarity and Matching of Neural Network Representationscited by 77
  14. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  15. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
  16. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  17. 2021On the Reproducibility of Neural Network Predictionscited by 28
  18. 2022Model Stability with Continuous Data Updatescited by 9
  19. 2021How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Taskcited by 5
  20. 2022Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clippingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 2; 2026: 2.

7 papers.

  1. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  2. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  3. 2024Punctuation Restoration Improves Structure Understanding Without Supervisioncited by 3
  4. 2023FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamicscited by 2
  5. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  6. 2026The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instrumentscited by 1
  7. 2026Variation is the Norm: Embracing Sociolinguistics in NLPcited by 0