mlsanityData to 7 Oct 2026

Transformers, Contextualism, and Polysemy

arXiv 2404.09577 ↗First posted 15 Apr 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The transformer architecture, introduced by Vaswani et al. (2017), is at the heart of the remarkable recent progress in the development of language models, including widely-used chatbots such as Chat-GPT and Claude. In this paper, I argue that we can extract from the way the transformer architecture works a theory of the relationship between context and meaning. I call this the transformer theory, and I argue that it is novel with regard to two related philosophical debates: the contextualism debate regarding the extent of context-sensitivity across natural language, and the polysemy debate regarding how polysemy should be captured within an account of word meaning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 14 of its 48 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

14 of 48 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  7. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  8. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  9. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  10. 2023The Vector Grounding Problemcited by 39
  11. 2021Dodrio: Exploring Transformer Models with Interactive Visualizationcited by 28
  12. 2023Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddingscited by 19
2 more
  1. 2020Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledgecited by 18
  2. 2019Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddingscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguitycited by 3
  2. 2025Word Meanings in Transformer Language Modelscited by 2
  3. 2025How do Transformer Embeddings Represent Compositions? A Functional Analysiscited by 1
  4. 2026Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summariescited by 0
  5. 2026Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropycited by 0