mlsanityData to 14 Oct 2026

Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models

arXiv 2404.01992 ↗First posted 2 Apr 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects or objects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

35 of 44 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  12. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
23 more
  1. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 324
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  3. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  4. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  5. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  6. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  7. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  8. 2019Assessing BERT's Syntactic Abilitiescited by 192
  9. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  10. 2020How Context Affects Language Models' Factual Predictionscited by 181
  11. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  12. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  13. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  14. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  15. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  16. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  17. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  18. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  19. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  20. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
  21. 2019Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representationscited by 24
  22. 2022KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Modelscited by 6
  23. 2019Shallow Syntax in Deep Watercited by 3

Built on by

Not cited yet.