mlsanityData to 7 Oct 2026

COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models

arXiv 2210.01963 ↗First posted 5 Oct 2022cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog) -- i.e. demonstrate property inheritance. In this paper, we present COMPS, a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal that they can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the basis of nuanced knowledge representations. Furthermore, we find that PLMs can demonstrate behavior consistent with property inheritance to a great extent, but fail in the presence of distracting information, which decreases the performance of many models, sometimes even below chance. This lack of robustness in demonstrating simple reasoning raises important questions about PLMs' capacity to make correct inferences even when they appear to possess the prerequisite knowledge.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 45 of its 90 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

45 of 90 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019Language Models as Knowledge Bases?cited by 1,922
33 more
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  5. 2019Designing and Interpreting Probes with Control Taskscited by 608
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  7. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2022Can language models learn from explanations in context?cited by 248
  11. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  12. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2022Language models show human-like content effects on reasoning taskscited by 171
  15. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  16. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  17. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  18. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  19. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  20. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  21. 2022Meaning without reference in large language modelscited by 76
  22. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  23. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  24. 2020Word meaning in minds and machinescited by 51
  25. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
  26. 2021Do language models learn typicality judgments from text?cited by 32
  27. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  28. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
  29. 2019Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representationscited by 24
  30. 2020Constructing Taxonomies from Pretrained Language Modelscited by 21
  31. 2017Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaningcited by 19
  32. 2022A Property Induction Framework for Neural Language Modelscited by 8
  33. 2018Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tellcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  3. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  4. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  5. 2022Language model acceptability judgements are not always robust to contextcited by 22
  6. 2024The Factuality of Large Language Models in the Legal Domaincited by 10
  7. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  8. 2025Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Modelscited by 2
  9. 2025AI-enhanced semantic feature norms for 786 conceptscited by 2
  10. 2026From Chains to DAGs: Probing the Graph Structure of Reasoning in LLMscited by 2
  11. 2024ACCORD: Closing the Commonsense Measurability Gapcited by 1
  12. 2025Bridging Reasoning to Learning: Unmasking Illusions using Complexity Out of Distribution Generalizationcited by 1
3 more
  1. 2025Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweightingcited by 1
  2. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0
  3. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0