mlsanityData to 7 Oct 2026

Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach

arXiv 2209.06995 ↗First posted 15 Sep 2022cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Large Language Models have demonstrated remarkable few-shot performance, but the performance can be sensitive to the selection of few-shot instances. We propose PATRON, a new method that uses prompt-based uncertainty estimation for data selection for pre-trained language model fine-tuning under cold-start scenarios, i.e., no initial labeled data are available. In PATRON, we design (1) a prompt-based uncertainty propagation approach to estimate the importance of data points and (2) a partition-then-rewrite (PTR) strategy to promote sample diversity when querying for annotations. Experiments on six text classification datasets show that PATRON outperforms the strongest cold-start data selection baselines by up to 6.9%. Besides, with 128 labels only, PATRON achieves 91.0% and 92.1% of the fully supervised performance based on vanilla fine-tuning and prompt-based learning respectively. Our implementation of PATRON is available at urlhttps://github.com/yueyu1030/Patron.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 53 of its 78 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

53 of 78 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
38 more
  1. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  5. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  6. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  7. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  8. 2018Learning to Perform Local Rewriting for Combinatorial Optimizationcited by 183
  9. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  10. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  11. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  12. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  13. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  14. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  15. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  16. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  17. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  18. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  19. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  20. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  21. 2022A Survey on Programmatic Weak Supervisioncited by 66
  22. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  23. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  24. 2020SeqMix: Augmenting Active Sequence Labeling via Sequence Mixupcited by 52
  25. 2018Weakly-Supervised Hierarchical Text Classificationcited by 42
  26. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  27. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  28. 2018Active Learning with Partial Feedbackcited by 27
  29. 2021Cartography Active Learningcited by 26
  30. 2021On Training Instance Selection for Few-Shot Neural Text Generationcited by 25
  31. 2020On Initial Pools for Deep Active Learningcited by 18
  32. 2020Active Sentence Learning by Adversarial Uncertainty Sampling in Discrete Spacecited by 17
  33. 2022Prompt Tuning for Discriminative Pre-trained Language Modelscited by 17
  34. 2022PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learningcited by 14
  35. 2022Cluster & Tune: Boost Cold Start Performance in Text Classificationcited by 13
  36. 2021Adapting Coreference Resolution Models through Active Learningcited by 11
  37. 2022Active Few-Shot Learning with FASLcited by 11
  38. 2021LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Frameworkcited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 12; 2024: 2; 2025: 1.

16 papers.

Sort:

By area:

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  3. 2024Active Learning for Vision-Language Modelscited by 15
  4. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  5. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  6. 2023Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learningcited by 9
  7. 2023R-Mixup: Riemannian Mixup for Biological Networkscited by 8
  8. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  9. 2022Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructionscited by 7
  10. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  11. 2023Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervisioncited by 4
  12. 2023Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learningcited by 4
4 more
  1. 2024A Comprehensive Review of Machine Learning Advances on Data Change: A Cross-Field Perspectivecited by 1
  2. 2023Local Boosting for Weakly-Supervised Learningcited by 0
  3. 2023G-STO: Sequential Main Shopping Intention Detection via Graph-Regularized Stochastic Transformercited by 0
  4. 2025Small sample-based adaptive text classification through iterative and contrastive description refinementcited by 0