Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers
Not on the map. Why?
Abstract (arXiv)
Distributional semantics is the linguistic theory that a word's meaning can be derived from its distribution in natural language (i.e., its use). Language models are commonly viewed as an implementation of distributional semantics, as they are optimized to capture the statistical features of natural language. It is often argued that distributional semantics models should excel at capturing graded/vague meaning based on linguistic conventions, but struggle with truth-conditional reasoning and symbolic processing. We evaluate this claim with a case study on vague (e.g. "many") and exact (e.g. "more than half") quantifiers. Contrary to expectations, we find that, across a broad range of models of various types, LLMs align more closely with human judgements on exact quantifiers versus vague ones. These findings call for a re-evaluation of the assumptions underpinning what distributional semantics models are, as well as what they can capture.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 46 references that are in the map. First posted on arXiv 17 Oct 2024.
Builds on
20 of 46 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
- 2022Binding Language Models in Symbolic Languagescited by 228
- 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
8 more
- 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
- 2019Distributional Semantics and Linguistic Theorycited by 53
- 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
- 2015Combining Language and Vision with a Multimodal Skip-gram Modelcited by 28
- 2022Improving negation detection with negation-focused pre-trainingcited by 14
- 2015Representing Meaning with a Combination of Logical and Distributional Modelscited by 7
- 2023Probing Quantifier Comprehension in Large Language Models: Another Example of Inverse Scalingcited by 5
- 2022Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifierscited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.