mlsanityData to 14 Oct 2026

Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords

arXiv 2109.11491 ↗First posted 23 Sep 2021cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses. By inducing a contextualized "pseudoword" as a stand-in for a static embedding in the input layer, and then performing masked prediction of a word in the sentence, we are able to investigate the geometry of the BERT-space in a controlled manner around individual instances. Using our method on a set of carefully constructed sentences targeting ambiguous English words, we find substantial regularity in the contextualized space, with regions that correspond to distinct word senses; but between these regions there are occasionally "sense voids" -- regions that do not correspond to any intelligible sense.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 23 Sep 2021.

Builds on

28 of 42 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Attention is not Explanationcited by 929
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  6. 2020Quantifying Attention Flow in Transformerscited by 775
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  10. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  11. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 216
16 more
  1. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  2. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  3. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  4. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  5. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  6. 2019Probing What Different NLP Tasks Teach Machines about Function Word Comprehensioncited by 44
  7. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 40
  8. 2019Linking artificial and human neural representations of languagecited by 37
  9. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  10. 2021DirectProbe: Studying Representations without Classifierscited by 25
  11. 2018Comprehensive Supersense Disambiguation of English Prepositions and Possessivescited by 20
  12. 2020Quantifying the Contextualization of Word Representations with Semantic Class Probingcited by 18
  13. 2020BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representationscited by 15
  14. 2021Mediators in Determining what Processing BERT Performs Firstcited by 14
  15. 2021RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)cited by 10
  16. 2020Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddingscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 0; 2025: 1; 2026: 1.

9 papers.

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  2. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  3. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  4. 2023Do PLMs Know and Understand Ontological Knowledge?cited by 13
  5. 2023AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suitecited by 6
  6. 2025Construction Identification and Disambiguation Using BERT: A Case Study of NPNcited by 4
  7. 2022Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in contextcited by 3
  8. 2022Reconstruction Probingcited by 2
  9. 2026Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMscited by 0