mlsanityData to 7 Oct 2026

The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations

arXiv 2402.14616 ↗First posted 22 Feb 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

When deriving contextualized word representations from language models, a decision needs to be made on how to obtain one for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represent these words with a single vector, and are these representations of worse quality than those of in-vocabulary words? We carry out an intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words. Our analysis reveals, among other interesting findings, that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words. Their similarity values, however, must be interpreted with caution.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 86 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

38 of 86 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,088
26 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  5. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  6. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  7. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  8. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  9. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  10. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  11. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  12. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  13. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 40
  14. 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
  15. 2021Explaining and Improving BERT Performance on Lexical Semantic Change Detectioncited by 33
  16. 2021DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languagescited by 25
  17. 2021AVocaDo: Strategy for Adapting Vocabulary to Downstream Domaincited by 20
  18. 2019BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performancecited by 18
  19. 2022BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languagescited by 14
  20. 2017High-risk learning: acquiring new word vectors from tiny datacited by 13
  21. 2019CoSimLex: A Resource for Evaluating Graded Word Similarity in Contextcited by 13
  22. 2022Breaking Character: Are Subwords Good Enough for MRLs After All?cited by 13
  23. 2018Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Modelscited by 11
  24. 2021Disambiguatory Signals are Stronger in Word-initial Positionscited by 9
  25. 2019Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutescited by 5
  26. 2018Unseen Word Representation by Aligning Heterogeneous Lexical Semantic Spacescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025Enhancing Lexicon-Based Text Embeddings with Large Language Modelscited by 6
  2. 2024Exploring Language Model Generalization in Low-Resource Extractive QAcited by 2
  3. 2025KinyaColBERT: A Lexically Grounded Retrieval Model for Low-Resource Retrieval-Augmented Generationcited by 2
  4. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  5. 2025One Word Is Not Enough: Simple Prompts Improve Word Embeddingscited by 1
  6. 2026Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probingcited by 0