mlsanityData to 7 Oct 2026

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

arXiv 1911.03343 ↗First posted 8 Nov 2019cs.CLcited by 240

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated ("Birds cannot [MASK]") and non-negated ("Birds can [MASK]") cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add "misprimes" to cloze questions ("Talk? Birds can [MASK]"). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

Key facts. Cited by 240 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 56; 2023: 58; 2024: 40; 2025: 26; 2026: 24.

240 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  5. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  8. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  10. 2022Large Language Models with Controllable Working Memorycited by 169
  11. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  12. 2022A Review on Language Models as Knowledge Basescited by 156
38 more
  1. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  2. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  3. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  6. 2024To Believe or Not to Believe Your LLMcited by 120
  7. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  8. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  9. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  10. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  11. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  12. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  13. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  14. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  15. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  16. 2022ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classificationcited by 66
  17. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  18. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  19. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  20. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  21. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 45
  22. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  23. 2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
  24. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  25. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  26. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  27. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  28. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  29. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  30. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  31. 2023Consistency Analysis of ChatGPTcited by 37
  32. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  33. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  34. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  35. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  36. 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
  37. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  38. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32

50 most cited of 240 All 240 (JSON)