mlsanityData to 7 Oct 2026

Do Prompt-Based Models Really Understand the Meaning of their Prompts?

arXiv 2109.01247 ↗First posted 2 Sep 2021cs.CLcited by 324

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompt templates manually written for natural language inference (NLI). We find that models learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively "good" prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2022). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models' impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions.

Key facts. Cited by 324 papers in the map (data through 7 Oct 2026). Builds on 30 of its 75 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

30 of 75 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2019How Can We Know What Language Models Know?cited by 953
18 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  7. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  8. 2021How Many Data Points is a Prompt Worth?cited by 197
  9. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  10. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  11. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  12. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  13. 2020Learning from Task Descriptionscited by 78
  14. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  15. 2020UnNatural Language Inferencecited by 68
  16. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  17. 2021What Context Features Can Transformer Language Models Use?cited by 49
  18. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 58; 2023: 106; 2024: 79; 2025: 38; 2026: 40.

324 papers.

Sort:

By area:

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
38 more
  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  2. 2022Using cognitive psychology to understand GPT-3cited by 389
  3. 2023Larger language models do in-context learning differentlycited by 386
  4. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  5. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  6. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  7. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  8. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  9. 2022Can language models learn from explanations in context?cited by 248
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  11. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  12. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  13. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  14. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  15. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  16. 2022Can Foundation Models Wrangle Your Data?cited by 173
  17. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  18. 2022Language models show human-like content effects on reasoning taskscited by 171
  19. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  20. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  21. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  22. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  23. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  24. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  25. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  26. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  27. 2023The Learnability of In-Context Learningcited by 126
  28. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  29. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 103
  30. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  31. 2023Modular Deep Learningcited by 82
  32. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  33. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  34. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 78
  35. 2023In-Context Learning through the Bayesian Prismcited by 78
  36. 2023Eight Things to Know about Large Language Modelscited by 73
  37. 2023Symbol tuning improves in-context learning in language modelscited by 72
  38. 2023Unleashing the potential of prompt engineering for large language modelscited by 72

50 most cited of 324 All 324 (JSON)