mlsanityData to 7 Oct 2026

Revisiting Few-sample BERT Fine-tuning

arXiv 2006.05987 ↗First posted 10 Jun 2020cs.CL cs.LGcited by 237

Not on the map. Why?

Abstract (arXiv)

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 80; 2023: 45; 2024: 30; 2025: 11; 2026: 13.

237 papers.

Sort:

By area:

  1. 2021GPT Understands, Toocited by 715
  2. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  3. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  4. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  7. 2021How Many Data Points is a Prompt Worth?cited by 197
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  9. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  10. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  11. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  12. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
38 more
  1. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  2. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  3. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  4. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  5. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  6. 2022Efficient Few-Shot Learning Without Promptscited by 135
  7. 2021Entailment as Few-Shot Learnercited by 122
  8. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  9. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  10. 2022Fusing finetuned models for better pretrainingcited by 110
  11. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  12. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  13. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  14. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  15. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  16. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
  17. 2021Comparing Test Sets with Item Response Theorycited by 55
  18. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  19. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  20. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  21. 2022Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuningcited by 50
  22. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  23. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  24. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  25. 2021Learning to Retrieve Passages without Supervisioncited by 46
  26. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  27. 2022TransPolymer: a Transformer-based language model for polymer property predictionscited by 43
  28. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  29. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  30. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  31. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  32. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  33. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  34. 2022Deepfake Text Detection: Limitations and Opportunitiescited by 37
  35. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  36. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  37. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  38. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33

50 most cited of 237 All 237 (JSON)