mlsanityData to 14 Oct 2026

SuTRA : Structurally-Unified Tokenization with Root Awareness

arXiv 2608.18087 ↗First posted 5 Jun 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

29 of 57 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  11. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  12. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
17 more
  1. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  2. 2025SuperBPE: Space Travel for Language Modelscited by 63
  3. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  4. 2026PaperBanana: Automating Academic Illustration for AI Scientistscited by 26
  5. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  6. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  7. 2022TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizercited by 13
  8. 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
  9. 2022UniMorph 4.0: Universal Morphologycited by 11
  10. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 11
  11. 2024T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddingscited by 10
  12. 2023MorphPiece : A Linguistic Tokenizer for Large Language Modelscited by 9
  13. 2025MorphTok: Morphologically Grounded Tokenization for Indian Languagescited by 7
  14. 2024LBPE: Long-token-first Tokenization to Improve Large Language Modelscited by 5
  15. 2024BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiencycited by 4
  16. 2026SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizerscited by 2
  17. 2026Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiececited by 1

Built on by

Not cited yet.