mlsanityData to 14 Oct 2026

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

arXiv 2609.05333 ↗First posted 4 Sep 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

19 of 30 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  10. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  11. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  12. 2024Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semanticscited by 44
7 more
  1. 2025Exploring the Word Sense Disambiguation Capabilities of Large Language Modelscited by 10
  2. 2025Do Large Language Models Understand Word Senses?cited by 10
  3. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  4. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  5. 2024To Word Senses and Beyond: Inducing Concepts with Contextualized Language Modelscited by 3
  6. 2026In the LLM era, Word Sense Induction remains unsolvedcited by 2
  7. 2026Divergent large language model predictions from convergent representations in ambiguous word pairscited by 1

Built on by

Not cited yet.