mlsanityData to 7 Oct 2026

Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners

arXiv 2108.13161 ↗First posted 30 Aug 2021cs.CL cs.AI cs.CVcited by 115

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners without any prompt engineering. The main principle behind this approach involves reformulating potential natural language processing tasks into the task of a pre-trained language model and differentially optimizing the prompt template as well as the target label with backpropagation. Furthermore, the proposed approach can be: (i) Plugged to any pre-trained language models; (ii) Extended to widespread classification tasks. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves a better few-shot performance. Code is available in https://github.com/zjunlp/DART.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 48 of its 55 references that are in the map. First posted on arXiv 30 Aug 2021.

Builds on

48 of 55 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
36 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2021GPT Understands, Toocited by 715
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  9. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  10. 2021True Few-Shot Learning with Language Modelscited by 343
  11. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  12. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  13. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 271
  14. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  15. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  16. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  17. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  18. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  19. 2021How Many Data Points is a Prompt Worth?cited by 197
  20. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  21. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  22. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  23. 2021Entailment as Few-Shot Learnercited by 122
  24. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  25. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
  26. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  27. 2019OpenNRE: An Open and Extensible Toolkit for Neural Relation Extractioncited by 62
  28. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  29. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  30. 2019Few-shot Text Classification with Distributional Signaturescited by 58
  31. 2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
  32. 2019Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detectioncited by 54
  33. 2020Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Startcited by 44
  34. 2019Relation Adversarial Network for Low Resource Knowledge Graph Completioncited by 24
  35. 2020Bridging Text and Knowledge with Multi-Prototype Embedding for Few-Shot Relational Triple Extractioncited by 18
  36. 2019When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classificationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 33; 2023: 36; 2024: 31; 2025: 9; 2026: 2.

115 papers.

Sort:

By area:

  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  5. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  6. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  7. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  8. 2022A Review on Language Models as Knowledge Basescited by 156
  9. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  10. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  11. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  12. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
38 more
  1. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  2. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  3. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  4. 2022From Discrimination to Generation: Knowledge Graph Completion with Generative Transformercited by 45
  5. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  6. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  7. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 39
  8. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  9. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  10. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36
  11. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  12. 2023BLESS: Benchmarking Large Language Models on Sentence Simplificationcited by 34
  13. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 32
  14. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  15. 2023Semantic Prompt for Few-Shot Image Recognitioncited by 29
  16. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 29
  17. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  18. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  19. 2022Ontology-enhanced Prompt-tuning for Few-shot Learningcited by 24
  20. 2021SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysiscited by 23
  21. 2023Editing Language Model-based Knowledge Graph Embeddingscited by 23
  22. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  23. 2022DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Populationcited by 17
  24. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  25. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  26. 2024StraGo: Harnessing Strategic Guidance for Prompt Optimizationcited by 16
  27. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  28. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  29. 2023SPELL: Semantic Prompt Evolution based on a LLMcited by 14
  30. 2023Automatic Engineering of Long Promptscited by 14
  31. 2024Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Wayscited by 14
  32. 2022Generative Prompt Tuning for Relation Classificationcited by 13
  33. 2023Assisting Language Learners: Automated Trans-Lingual Definition Generation via Contrastive Prompt Learningcited by 13
  34. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  35. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  36. 2024Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognitioncited by 11
  37. 2022Robot Task Planning and Situation Handling in Open Worldscited by 10
  38. 2023Scalable Prompt Generation for Semi-supervised Learning with Language Modelscited by 10

50 most cited of 115 All 115 (JSON)