mlsanityData to 14 Oct 2026

FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding

arXiv 2109.12742 ↗First posted 27 Sep 2021cs.CL cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

27 of 46 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
15 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  4. 2021GPT Understands, Toocited by 716
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  7. 2021True Few-Shot Learning with Language Modelscited by 344
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  9. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  10. 2021Free Lunch for Few-shot Learning: Distribution Calibrationcited by 165
  11. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  12. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  13. 2019Learning and Evaluating General Linguistic Intelligencecited by 79
  14. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  15. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 6; 2024: 0; 2025: 0; 2026: 1.

17 papers.

Sort:

  1. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  2. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 163
  3. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  4. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
  5. 2022MEAL: Stable and Active Learning for Few-Shot Promptingcited by 20
  6. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  7. 2023Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysiscited by 11
  8. 2021CLUES: Few-Shot Learning Evaluation in Natural Language Understandingcited by 9
  9. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  10. 2023Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understandingcited by 8
  11. 2023A Challenging Benchmark for Low-Resource Learningcited by 5
  12. 2022Zero-Label Prompt Selectioncited by 4
5 more
  1. 2023Sociocultural knowledge is needed for selection of shots in hate speech detection taskscited by 1
  2. 2022What are the best systems? New perspectives on NLP Benchmarkingcited by 0
  3. 2022Rethinking the Setting of Semi-supervised Learning on Graphscited by 0
  4. 2023How to Solve Few-Shot Abusive Content Detection Using the Data We Actually Havecited by 0
  5. 2026Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessmentcited by 0