mlsanityData to 7 Oct 2026

Are Prompt-based Models Clueless?

arXiv 2205.09295 ↗First posted 19 May 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Finetuning large pre-trained language models with a task-specific head has advanced the state-of-the-art on many natural language understanding benchmarks. However, models with a task-specific head require a lot of training data, making them susceptible to learning and exploiting dataset-specific superficial cues that do not generalize to other datasets. Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective. Therefore, it is expected that few-shot prompt-based models do not exploit superficial cues. This paper presents an empirical examination of whether few-shot prompt-based models also exploit superficial cues. Analyzing few-shot prompt-based models on MNLI, SNLI, HANS, and COPA has revealed that prompt-based models also exploit superficial cues. While the models perform well on instances with superficial cues, they often underperform or only marginally outperform random accuracy on instances without superficial cues.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

24 of 28 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
12 more
  1. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  3. 2019Attention is not not Explanationcited by 629
  4. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  5. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  6. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  7. 2021How Many Data Points is a Prompt Worth?cited by 197
  8. 2020Adversarial Filters of Dataset Biasescited by 159
  9. 2019Towards Debiasing Fact Verification Modelscited by 143
  10. 2018What Makes Reading Comprehension Questions Easier?cited by 59
  11. 2019When Choosing Plausible Alternatives, Clever Hans can be Clevercited by 39
  12. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 7; 2024: 1.

11 papers.

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25
  5. 2023Evaluating the Robustness of Discrete Promptscited by 18
  6. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  7. 2023Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understandingcited by 8
  8. 2023Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Trackingcited by 6
  9. 2022Continuous Prompt Tuning Based Textual Entailment Model for E-commerce Entity Typingcited by 5
  10. 2022Explanation Regeneration via Information Bottleneckcited by 4
  11. 2023Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuningcited by 1