mlsanityData to 7 Oct 2026

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

arXiv 2107.13586 ↗First posted 28 Jul 2021cs.CL cs.AI cs.LGcited by 2,222

Not on the map. Why?

Cited by 123 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

Key facts. Cited by 2,222 papers in the map (data through 7 Oct 2026). Builds on 155 of its 224 references that are in the map. First posted on arXiv 28 Jul 2021.

Builds on

155 of 224 references.

References · 155

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  13. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  14. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  15. 2019Language Models as Knowledge Bases?cited by 1,922
  16. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  17. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  18. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  19. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  20. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  21. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  22. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  23. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  24. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  25. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  26. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  27. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  28. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  29. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  30. 2019How Can We Know What Language Models Know?cited by 953
  31. 2017A Closer Look at Memorization in Deep Networkscited by 936
  32. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  33. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  34. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  35. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  36. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  37. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  38. 2021GPT Understands, Toocited by 715

50 most cited of 155 All 155 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 59; 2022: 430; 2023: 732; 2024: 518; 2025: 313; 2026: 170.

2,222 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2022Visual Prompt Tuningcited by 1,674
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2021Learning to Prompt for Continual Learningcited by 938
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  6. 2022A Survey on In-context Learningcited by 717
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  12. 2022PAL: Program-aided Language Modelscited by 635
  13. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  14. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  15. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  16. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  17. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  18. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  20. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  21. 2024Large Language Models: A Surveycited by 506
  22. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  23. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  24. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  25. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  26. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  27. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  28. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  29. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  30. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  31. 2023Cognitive Architectures for Language Agentscited by 422
  32. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  33. 2022Multimodal Learning with Transformers: A Surveycited by 399
  34. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  35. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  36. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  37. 2023GPTScore: Evaluate as You Desirecited by 357
  38. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355

50 most cited of 2,222 All 2,222 (JSON)