mlsanityData to 14 Oct 2026

Tokenization with Factorized Subword Encoding

arXiv 2306.07764 ↗First posted 13 Jun 2023cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, language models have become increasingly larger and more complex. However, the input representations for these models continue to rely on simple and greedy subword tokenization methods. In this paper, we propose a novel tokenization method that factorizes subwords onto discrete triplets using a VQ-VAE model. The effectiveness of the proposed tokenization method, referred to as the Factorizer, is evaluated on language modeling and morpho-syntactic tasks for 7 diverse languages. Results indicate that this method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

31 of 50 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  12. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
19 more
  1. 2020Jukebox: A Generative Model for Musiccited by 542
  2. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  3. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  5. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  7. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 104
  8. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  9. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  10. 2020Hierarchical Quantized Autoencoderscited by 57
  11. 2022Language Modelling with Pixelscited by 55
  12. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  13. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  14. 2017An optimal transportation approach for assessing almost stochastic ordercited by 33
  15. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  16. 2018How Much Does Tokenization Affect Neural Machine Translation?cited by 26
  17. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  18. 2019UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Mergingcited by 15
  19. 2017From Characters to Words to in Between: Do We Capture Morphology?cited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 3.

4 papers.

  1. 2026Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Featurescited by 1
  2. 2024How Important Is Tokenization in French Medical Masked Language Models?cited by 0
  3. 2026BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysiscited by 0
  4. 2026The Functionalizer: Lossless Functional Decomposition for Subword Tokenizationcited by 0