# A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning

arXiv: https://arxiv.org/abs/2204.10815 · HTML: https://mlsanity.com/paper/2204.10815/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Improving Multilingual Models with Language-Clustered Vocabularies](https://mlsanity.com/paper/2010.12777/) (2020)
- [Neural Cross-Lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1808.09861/) (2018)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [Character-based Neural Machine Translation](https://mlsanity.com/paper/1603.00810/) (2016)
- [Character-Level Language Modeling with Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1609.03777/) (2016)
- [Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality](https://mlsanity.com/paper/2010.12730/) (2020)
- [Character Eyes: Seeing Language through Character-Level Taggers](https://mlsanity.com/paper/1903.05041/) (2019)

## Built on by (most cited first, 12 of 12)

- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models](https://mlsanity.com/paper/2505.14172/) (2025)
- [Beyond Literal Token Overlap: Token Alignability for Multilinguality](https://mlsanity.com/paper/2502.06468/) (2025)
- [Learn Your Tokens: Word-Pooled Tokenization for Language Modeling](https://mlsanity.com/paper/2310.11628/) (2023)
- [FLEXITOKENS: Flexible Tokenization for Evolving Language Models](https://mlsanity.com/paper/2507.12720/) (2025)
- [From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding](https://mlsanity.com/paper/2305.14571/) (2023)
- [TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior](https://mlsanity.com/paper/2512.20757/) (2025)
- [MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling](https://mlsanity.com/paper/2212.07284/) (2022)
- [CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models](https://mlsanity.com/paper/2403.02745/) (2024)
- [LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation](https://mlsanity.com/paper/2603.22629/) (2026)

## Abstract (arXiv)

Subword tokenization is a commonly used input pre-processing step in most recent NLP models. However, it limits the models' ability to leverage end-to-end task learning. Its frequency-based vocabulary creation compromises tokenization in low-resource languages, leading models to produce suboptimal representations. Additionally, the dependency on a fixed vocabulary limits the subword models' adaptability across languages and domains. In this work, we propose a vocabulary-free neural tokenizer by distilling segmentation information from heuristic-based subword tokenization. We pre-train our character-based tokenizer by processing unique words from multilingual corpus, thereby extensively increasing word diversity across languages. Unlike the predefined and fixed vocabularies in subword methods, our tokenizer allows end-to-end task learning, resulting in optimal task-specific tokenization. The experimental results show that replacing the subword tokenizer with our neural tokenizer consistently improves performance on multilingual (NLI) and code-switching (sentiment analysis) tasks, with larger gains in low-resource languages. Additionally, our neural tokenizer exhibits a robust performance on downstream tasks when adversarial noise is present (typos and misspelling), further increasing the initial improvements over statistical subword tokenizers.