mlsanityData to 7 Oct 2026

Incorporating Context into Subword Vocabularies

arXiv 2210.07095 ↗First posted 13 Oct 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Most current popular subword tokenizers are trained based on word frequency statistics over a corpus, without considering information about co-occurrence or context. Nevertheless, the resulting vocabularies are used in language models' highly contextualized settings. We present SaGe, a tokenizer that tailors subwords for their downstream use by baking in the contextualized signal at the vocabulary creation phase. We show that SaGe does a better job than current widespread tokenizers in keeping token contexts cohesive, while not incurring a large price in terms of encoding efficiency or domain robustness. SaGe improves performance on English GLUE classification tasks as well as on NER, and on Inference and NER in Turkish, demonstrating its robustness to language properties such as morphological exponence and agglutination.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

33 of 48 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  12. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
21 more
  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  2. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  3. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  4. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  5. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  6. 2021How to Train BERT with an Academic Budgetcited by 80
  7. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  8. 2019CrossWeigh: Training Named Entity Tagger from Imperfect Annotationscited by 60
  9. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  10. 2022Language Modelling with Pixelscited by 53
  11. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  12. 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
  13. 2022What do tokens know about their characters and how do they know it?cited by 36
  14. 2014POLYGLOT-NER: Massive Multilingual Named Entity Recognitioncited by 32
  15. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  16. 2021Joint Optimization of Tokenization and Downstream Modelcited by 15
  17. 2018Optimizing Segmentation Granularity for Neural Machine Translationcited by 14
  18. 2022Breaking Character: Are Subwords Good Enough for MRLs After All?cited by 13
  19. 2021Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languagescited by 8
  20. 2022MaxMatch-Dropout: Subword Regularization for WordPiececited by 8
  21. 2020Subword Sampling for Low Resource Word Alignmentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 4; 2026: 3.

11 papers.

  1. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  2. 2025ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendationcited by 18
  3. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  4. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  5. 2024Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methodscited by 6
  6. 2026MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignmentcited by 5
  7. 2024Infusing clinical knowledge into tokenisers for language modelscited by 2
  8. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
  9. 2023PatternGPT :A Pattern-Driven Framework for Large Language Model Text Generationcited by 1
  10. 2025UniNet: A Unified Multi-granular Traffic Modeling Framework for Network Securitycited by 1
  11. 2026Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruningcited by 0