# ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality

arXiv: https://arxiv.org/abs/2110.06376 · HTML: https://mlsanity.com/paper/2110.06376/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [Combining Language and Vision with a Multimodal Skip-gram Model](https://mlsanity.com/paper/1501.02598/) (2015)
- [HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment](https://mlsanity.com/paper/1608.02117/) (2016)
- [Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaning](https://mlsanity.com/paper/1705.11168/) (2017)
- [Data Augmentation for Hypernymy Detection](https://mlsanity.com/paper/2005.01854/) (2020)
- [Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment](https://mlsanity.com/paper/1605.05433/) (2016)

## Built on by (most cited first, 12 of 12)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023)
- [Modelling Commonsense Properties using Pre-Trained Bi-Encoders](https://mlsanity.com/paper/2210.02771/) (2022)
- [Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution](https://mlsanity.com/paper/2410.17482/) (2024)
- [Is "My Favorite New Movie" My Favorite Movie? Probing the Understanding of Recursive Noun Phrases](https://mlsanity.com/paper/2112.08326/) (2021)
- [Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain](https://mlsanity.com/paper/2304.14745/) (2023)
- [Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction](https://mlsanity.com/paper/2210.12905/) (2022)
- [Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks](https://mlsanity.com/paper/2302.06942/) (2023)
- [3D-EX : A Unified Dataset of Definitions and Dictionary Examples](https://mlsanity.com/paper/2308.03043/) (2023)
- [What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies](https://mlsanity.com/paper/2310.14793/) (2023)
- [CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation](https://mlsanity.com/paper/2508.04494/) (2025)

## Abstract (arXiv)

Large scale language models encode rich commonsense knowledge acquired through exposure to massive data during pre-training, but their understanding of entities and their semantic properties is unclear. We probe BERT (Devlin et al., 2019) for the properties of English nouns as expressed by adjectives that do not restrict the reference scope of the noun they modify (as in "red car"), but instead emphasise some inherent aspect ("red strawberry"). We base our study on psycholinguistics datasets that capture the association strength between nouns and their semantic features. We probe BERT using cloze tasks and in a classification setting, and show that the model has marginal knowledge of these features and their prevalence as expressed in these datasets. We discuss factors that make evaluation challenging and impede drawing general conclusions about the models' knowledge of noun properties. Finally, we show that when tested in a fine-tuning setting addressing entailment, BERT successfully leverages the information needed for reasoning about the meaning of adjective-noun constructions outperforming previous methods.