The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
Not on the map. Why?
Abstract (arXiv)
Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and H'ello) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area. We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose. These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 7 Jul 2026.
Builds on
14 of 32 references.
References · 14
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
- 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
- 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
- 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
- 2019Naver Labs Europe's Systems for the WMT19 Machine Translation Robustness Taskcited by 23
- 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 12
- 2024Don't Touch My Diacriticscited by 7
2 more
- 2025Tokens with Meaning: A Hybrid Tokenization Approach for Turkishcited by 7
- 2023Tokenization with Factorized Subword Encodingcited by 4
Built on by
Not cited yet.