mlsanityData to 7 Oct 2026

A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness

arXiv 2312.01082 ↗First posted 2 Dec 2023cs.LG cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Learning with limited labelled data, such as prompting, in-context learning, fine-tuning, meta-learning or few-shot learning, aims to effectively train a model using only a small amount of labelled samples. However, these approaches have been observed to be excessively sensitive to the effects of uncontrolled randomness caused by non-determinism in the training process. The randomness negatively affects the stability of the models, leading to large variances in results across training runs. When such sensitivity is disregarded, it can unintentionally, but unfortunately also intentionally, create an imaginary perception of research progress. Recently, this area started to attract research attention and the number of relevant studies is continuously growing. In this survey, we provide a comprehensive overview of 415 papers addressing the effects of randomness on the stability of learning with limited labelled data. We distinguish between four main tasks addressed in the papers (investigate/evaluate; determine; mitigate; benchmark/compare/report randomness effects), providing findings for each one. Furthermore, we identify and discuss seven challenges and open problems together with possible directions to facilitate further research. The ultimate goal of this survey is to emphasise the importance of this growing research area, which so far has not received an appropriate level of attention, and reveal impactful directions for future research.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 123 of its 203 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

123 of 203 references.

References · 123

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2016Matching Networks for One Shot Learningcited by 2,246
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2023Whose Opinions Do Language Models Reflect?cited by 759
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  14. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  15. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  16. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  17. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  18. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  19. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  20. 2021True Few-Shot Learning with Language Modelscited by 343
  21. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  22. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  23. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  24. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  25. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  26. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  27. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  28. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  29. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  30. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  31. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  32. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  33. 2022Active Example Selection for In-Context Learningcited by 195
  34. 2022In-context Examples Selection for Machine Translationcited by 172
  35. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  36. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  37. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  38. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145

50 most cited of 123 All 123 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 1.

14 papers.

Sort:

By area:

  1. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 33
  2. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 17
  3. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  4. 2024Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classificationcited by 7
  5. 2024EBES: Easy Benchmarking for Event Sequencescited by 6
  6. 2024Automatic Combination of Sample Selection Strategies for Few-Shot Learningcited by 4
  7. 2024A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Modelscited by 3
  8. 2024Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolationcited by 2
  9. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classificationcited by 2
  10. 2024On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choicescited by 1
  11. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Setscited by 1
  12. 2025Gibbs randomness-compression propositioncited by 0
2 more
  1. 2025Color as the Impetus: Transforming Few-Shot Learnercited by 0
  2. 2025Improved Sub-Visible Particle Classification in Flow Imaging Microscopy via Generative AI-Based Image Synthesiscited by 0