mlsanityData to 7 Oct 2026

Learning New Tasks from a Few Examples with Soft-Label Prototypes

arXiv 2210.17437 ↗First posted 31 Oct 2022cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing approaches to few-shot learning in NLP rely on large language models (LLMs) and/or fine-tuning of these to generalise on out-of-distribution data. In this work, we propose a novel few-shot learning approach based on soft-label prototypes (SLPs) designed to collectively capture the distribution of different classes across the input domain space. We focus on learning previously unseen NLP tasks from very few examples (4, 8, 16) per class and experimentally demonstrate that our approach achieves superior performance on the majority of tested tasks in this data-lean setting while being highly parameter efficient. We also show that our few-shot adaptation method can be integrated into more generalised learning settings, primarily meta-learning, to yield superior performance against strong baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 85 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

50 of 85 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2018Neural Network Acceptability Judgmentscited by 952
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2019On the Convergence of Adam and Beyondcited by 834
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  13. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  14. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  15. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  16. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  17. 2018How to train your MAMLcited by 306
  18. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  19. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  20. 2018Bayesian Model-Agnostic Meta-Learningcited by 194
  21. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  22. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  23. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  24. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  25. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  26. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  27. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  28. 2020k-Nearest Neighbour Classifiers: 2nd Edition (with Python examples)cited by 56
  29. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  30. 2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
  31. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  32. 2021Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learningcited by 21
  33. 2020'Less Than One'-Shot Learning: Learning N Classes From M<N Samplescited by 15
  34. 2021Multilingual and cross-lingual document classification: A meta-learning approachcited by 14
  35. 2020Learning to Learn to Disambiguate: Meta-Learning for Few-Shot Word Sense Disambiguationcited by 13
  36. 2021Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsingcited by 10
  37. 2022Can Humans Do Less-Than-One-Shot Learning?cited by 7
  38. 2021One Line To Rule Them All: Generating LO-Shot Soft-Label Prototypescited by 4

Built on by

Not cited yet.