Lost in Space Marking
Not on the map. Why?
Abstract (arXiv)
We look at a decision taken early in training a subword tokenizer, namely whether it should be the word-initial token that carries a special mark, or the word-final one. Based on surface-level considerations of efficiency and cohesion, as well as morphological coverage, we find that a Unigram LM tokenizer trained on pre-tokenized English text is better off marking the word-initial token, while one trained on raw text benefits from marking word ends. Our findings generalize across domains.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 7 of its 13 references that are in the map. First posted on arXiv 2 Aug 2022.
Builds on
7 of 13 references.
References · 7
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
- 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
- 2019What Kind of Language Is Hard to Language-Model?cited by 50
- 2019Universal Dependency Parsing from Scratchcited by 46
Built on by
Citing papers per year: 2025: 2.
2 papers.