Using Distributional Principles for the Semantic Study of Contextual Language Models
Not on the map. Why?
Abstract (arXiv)
Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 49 references that are in the map. First posted on arXiv 23 Nov 2021.
Builds on
17 of 49 references.
References · 17
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2018Deep contextualized word representationscited by 2,285
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
- 2019Visualizing and Measuring the Geometry of BERTcited by 215
- 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
- 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
- 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
- 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
5 more
- 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
- 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 40
- 2019What do you mean, BERT? Assessing BERT as a Distributional Semantics Modelcited by 14
- 2019Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutescited by 5
- 2016CogALex-V Shared Task: ROOT18cited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.