mlsanityData to 7 Oct 2026

Improving and Simplifying Pattern Exploiting Training

arXiv 2103.11955 ↗First posted 22 Mar 2021cs.CL cs.AI cs.LGcited by 97

Not on the map. Why?

Abstract (arXiv)

Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data. Our code can be found at https://github.com/rrmenon10/ADAPET.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

17 of 31 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
5 more
  1. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  2. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  3. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  4. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  5. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 38; 2023: 18; 2024: 8; 2025: 2; 2026: 1.

97 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2021True Few-Shot Learning with Language Modelscited by 343
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  12. 2021Generating Datasets with Pretrained Language Modelscited by 192
38 more
  1. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  2. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  3. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  4. 2022Efficient Few-Shot Learning Without Promptscited by 135
  5. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  6. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  7. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  8. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  9. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  10. 2021Towards Zero-Label Language Learningcited by 76
  11. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  12. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  13. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  14. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  15. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  16. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  17. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  18. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  19. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  20. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  21. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  22. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  23. 2022CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domaincited by 31
  24. 2021CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systemscited by 30
  25. 2021Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Trainingcited by 28
  26. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  27. 2022Ontology-enhanced Prompt-tuning for Few-shot Learningcited by 24
  28. 2021NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Predictioncited by 23
  29. 2022What Makes Pre-trained Language Models Better Zero-shot Learners?cited by 23
  30. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  31. 2022MEAL: Stable and Active Learning for Few-Shot Promptingcited by 20
  32. 2021A Few More Examples May Be Worth Billions of Parameterscited by 19
  33. 2022Instance-aware Prompt Learning for Language Understanding and Generationcited by 18
  34. 2021FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understandingcited by 17
  35. 2022Contrastive Learning for Prompt-Based Few-Shot Language Learnerscited by 17
  36. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  37. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  38. 2021Distilling Relation Embeddings from Pre-trained Language Modelscited by 15

50 most cited of 97 All 97 (JSON)