mlsanityData to 7 Oct 2026

FLEX: Unifying Evaluation for Few-Shot NLP

arXiv 2107.07170 ↗First posted 15 Jul 2021cs.CL cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design. Consequently, the community does not know which techniques perform best or even if they outperform simple baselines. In response, we formulate the FLEX Principles, a set of requirements and best practices for unified, rigorous, valid, and cost-sensitive few-shot NLP evaluation. These principles include Sample Size Design, a novel approach to benchmark design that optimizes statistical accuracy and precision while keeping evaluation costs manageable. Following the principles, we release the FLEX benchmark, which includes four few-shot transfer settings, zero-shot evaluation, and a public leaderboard that covers diverse NLP tasks. In addition, we present UniFew, a prompt-based model for few-shot learning that unifies pretraining and finetuning prompt formats, eschewing complex machinery of recent prompt-based approaches in adapting downstream task formats to language model pretraining objectives. We demonstrate that despite simplicity, UniFew achieves results competitive with both popular meta-learning and prompt-based approaches.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 53 of its 82 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

53 of 82 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
38 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  6. 2019A Closer Look at Few-shot Classificationcited by 679
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2019Green AIcited by 620
  9. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  10. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  11. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  12. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  13. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  14. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  15. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  16. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  17. 2021True Few-Shot Learning with Language Modelscited by 343
  18. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  19. 2019A Baseline for Few-Shot Image Classificationcited by 298
  20. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  21. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  22. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  23. 2020With Little Power Comes Great Responsibilitycited by 138
  24. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  25. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  26. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  27. 2020Learning from Task Descriptionscited by 78
  28. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  29. 2020learn2learn: A Library for Meta-Learning Researchcited by 69
  30. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  31. 2019Few-shot Text Classification with Distributional Signaturescited by 58
  32. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  33. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  34. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  35. 2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
  36. 2020Meta-learning for Few-shot Natural Language Processing: A Surveycited by 42
  37. 2019A Theoretical Analysis of the Number of Shots in Few-Shot Learningcited by 39
  38. 2020Learning to Classify Intents and Slot Labels Given a Handful of Examplescited by 31

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 27; 2023: 18; 2024: 9; 2025: 3.

67 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  4. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  5. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  6. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  7. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  8. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  9. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  10. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  11. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  12. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
38 more
  1. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  2. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  3. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  4. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  5. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  6. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  7. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  8. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  9. 2022Zero-Shot Text Classification with Self-Trainingcited by 26
  10. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  11. 2023Building Efficient Universal Classifiers with Natural Language Inferencecited by 19
  12. 2021FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understandingcited by 17
  13. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  14. 2025Reliable Few-shot Learning under Dual Noisescited by 16
  15. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  16. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  17. 2023EZ-CLIP: Efficient Zeroshot Video Action Recognitioncited by 14
  18. 2023DETA: Denoised Task Adaptation for Few-Shot Learningcited by 13
  19. 2024Inductive Graph Alignment Prompt: Bridging the Gap between Graph Pre-training and Inductive Fine-tuning From Spectral Perspectivecited by 13
  20. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  21. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10
  22. 2022Few-Shot Document-Level Event Argument Extractioncited by 9
  23. 2023Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learningcited by 9
  24. 2022Pre-training to Match for Unified Low-shot Relation Extractioncited by 8
  25. 2022Why only Micro-F1? Class Weighting of Measures for Relation Classificationcited by 8
  26. 2023ZARA: Improving Few-Shot Self-Rationalization for Small Language Modelscited by 8
  27. 2023Weaker Than You Think: A Critical Look at Weakly Supervised Learningcited by 8
  28. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  29. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  30. 2022Czech Dataset for Cross-lingual Subjectivity Classificationcited by 5
  31. 2022Meta-Learning Online Control for Linear Dynamical Systemscited by 5
  32. 2023A Challenging Benchmark for Low-Resource Learningcited by 5
  33. 2023Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extractioncited by 5
  34. 2023Understanding Deep Neural Networks via Linear Separability of Hidden Layerscited by 4
  35. 2023RemovalNet: DNN Fingerprint Removal Attackscited by 4
  36. 2024Stronger Random Baselines for In-Context Learningcited by 4
  37. 2022Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligencecited by 3
  38. 2022FewFedWeight: Few-shot Federated Learning Framework across Multiple NLP Taskscited by 3

50 most cited of 67 All 67 (JSON)