# A comparative evaluation and analysis of three generations of Distributional Semantic Models

arXiv: https://arxiv.org/abs/2105.09825 · HTML: https://mlsanity.com/paper/2105.09825/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 85 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Compositional Languages Emerge in a Neural Iterated Learning Model](https://mlsanity.com/paper/2002.01365/) (2020)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Distributional Semantics and Linguistic Theory](https://mlsanity.com/paper/1905.01896/) (2019)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains](https://mlsanity.com/paper/1906.01539/) (2019)
- [Correlating neural and symbolic representations of language](https://mlsanity.com/paper/1905.06401/) (2019)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [What do you mean, BERT? Assessing BERT as a Distributional Semantics Model](https://mlsanity.com/paper/1911.05758/) (2019)
- [How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks](https://mlsanity.com/paper/1702.02170/) (2017)
- [Higher-order Comparisons of Sentence Encoder Representations](https://mlsanity.com/paper/1909.00303/) (2019)
- [The Effects of Data Size and Frequency Range on Distributional Semantic Models](https://mlsanity.com/paper/1609.08293/) (2016)
- [The Role of Context Types and Dimensionality in Learning Word Embeddings](https://mlsanity.com/paper/1601.00893/) (2016)
- [Don't Blame Distributional Semantics if it can't do Entailment](https://mlsanity.com/paper/1905.07356/) (2019)
- [Is Structure Necessary for Modeling Argument Expectations in Distributional Semantics?](https://mlsanity.com/paper/1710.00998/) (2017)

## Built on by (most cited first, 19 of 19)

- [A Psycholinguistic Analysis of BERT's Representations of Compounds](https://mlsanity.com/paper/2302.07232/) (2023)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [Understanding Natural Language Understanding Systems. A Critical Analysis](https://mlsanity.com/paper/2303.04229/) (2023)
- [Where's the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit](https://mlsanity.com/paper/2208.04749/) (2022)
- [The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect](https://mlsanity.com/paper/2505.15682/) (2025)
- [From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs](https://mlsanity.com/paper/2602.00628/) (2026)
- [ReFRAME or Remain: Unsupervised Lexical Semantic Change Detection with Frame Semantics](https://mlsanity.com/paper/2602.04514/) (2026)
- [Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse](https://mlsanity.com/paper/2402.09934/) (2024)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)
- [A vector logic for intensional formal semantics](https://mlsanity.com/paper/2602.02940/) (2026)
- [SynFlow: A Multidimensional Diachronic Semantic Analysis Toolkit](https://mlsanity.com/paper/2608.19472/) (2026)
- [Subdiffusive semantic evolution in Indo-European languages](https://mlsanity.com/paper/2209.04701/) (2022)
- [Montague semantics and modifier consistency measurement in neural language models](https://mlsanity.com/paper/2212.04310/) (2022)
- [Contextualized word senses: from attention to compositionality](https://mlsanity.com/paper/2312.00680/) (2023)
- [Values That Are Explicitly Present in Fairy Tales: Comparing Samples from German, Italian and Portuguese Traditions](https://mlsanity.com/paper/2402.08318/) (2024)
- [Identifying and interpreting non-aligned human conceptual representations using language modeling](https://mlsanity.com/paper/2403.06204/) (2024)
- [Large Language Models Preserve Semantic Isotopies in Story Continuations](https://mlsanity.com/paper/2510.04400/) (2025)
- [Early Language Learning via Spreading Activation and Category Exploration in Complex Networks](https://mlsanity.com/paper/2607.06258/) (2026)

## Abstract (arXiv)

Distributional semantics has deeply changed in the last decades. First, predict models stole the thunder from traditional count ones, and more recently both of them were replaced in many NLP applications by contextualized vectors produced by Transformer neural language models. Although an extensive body of research has been devoted to Distributional Semantic Model (DSM) evaluation, we still lack a thorough comparison with respect to tested models, semantic tasks, and benchmark datasets. Moreover, previous work has mostly focused on task-driven evaluation, instead of exploring the differences between the way models represent the lexical semantic space. In this paper, we perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. First of all, we investigate the performance of embeddings in several semantic tasks, carrying out an in-depth statistical analysis to identify the major factors influencing the behavior of DSMs. The results show that i.) the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous and ii.) static DSMs surpass contextualized representations in most out-of-context semantic tasks and datasets. Furthermore, we borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models. RSA reveals important differences related to the frequency and part-of-speech of lexical items.