# TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer

arXiv: https://arxiv.org/abs/2210.11753 · HTML: https://mlsanity.com/paper/2210.11753/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [FLAT: Chinese NER Using Flat-Lattice Transformer](https://mlsanity.com/paper/2004.11795/) (2020)
- [A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages](https://mlsanity.com/paper/2102.06551/) (2021)
- [Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit](https://mlsanity.com/paper/1809.01446/) (2018)
- [Evaluating Neural Morphological Taggers for Sanskrit](https://mlsanity.com/paper/2005.10893/) (2020)
- [Validation and Normalization of DCS corpus using Sanskrit Heritage tools to build a tagged Gold Corpus](https://mlsanity.com/paper/2005.06545/) (2020)
- [Sanskrit Sandhi Splitting using seq2(seq)^2](https://mlsanity.com/paper/1801.00428/) (2018)
- [Building a Word Segmenter for Sanskrit Overnight](https://mlsanity.com/paper/1802.06185/) (2018)

## Built on by (most cited first, 13 of 13)

- [One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks](https://mlsanity.com/paper/2409.13920/) (2024)
- [SanskritShala: A Neural Sanskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposes](https://mlsanity.com/paper/2302.09527/) (2023)
- [LEVOS: Leveraging Vocabulary Overlap with Sanskrit to Generate Technical Lexicons in Indian Languages](https://mlsanity.com/paper/2407.06331/) (2024)
- [Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents](https://mlsanity.com/paper/2505.19494/) (2025)
- [Still Not There: Can LLMs Outperform Smaller Task-Specific Seq2Seq Models on the Poetry-to-Prose Conversion Task?](https://mlsanity.com/paper/2511.08145/) (2025)
- [Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit](https://mlsanity.com/paper/2308.08807/) (2023)
- [DepNeCTI: Dependency-based Nested Compound Type Identification for Sanskrit](https://mlsanity.com/paper/2310.09501/) (2023)
- [Mapping Hymns and Organizing Concepts in the Rigveda: Quantitatively Connecting the Vedic Suktas](https://mlsanity.com/paper/2503.18226/) (2025)
- [Transformer-Enabled Diachronic Analysis of Vedic Sanskrit: Neural Methods for Quantifying Types of Language Change](https://mlsanity.com/paper/2512.05364/) (2025)
- [Rethinking Indic AI from a Lens of Cultural Heritage Preservation](https://mlsanity.com/paper/2607.06544/) (2026)
- [SuTRA : Structurally-Unified Tokenization with Root Awareness](https://mlsanity.com/paper/2608.18087/) (2026)
- [Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text](https://mlsanity.com/paper/2608.18437/) (2026)
- [Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition](https://mlsanity.com/paper/2609.12960/) (2026)

## Abstract (arXiv)

Sanskrit Word Segmentation (SWS) is essential in making digitized texts available and in deploying downstream tasks. It is, however, non-trivial because of the sandhi phenomenon that modifies the characters at the word boundaries, and needs special treatment. Existing lexicon driven approaches for SWS make use of Sanskrit Heritage Reader, a lexicon-driven shallow parser, to generate the complete candidate solution space, over which various methods are applied to produce the most valid solution. However, these approaches fail while encountering out-of-vocabulary tokens. On the other hand, purely engineering methods for SWS have made use of recent advances in deep learning, but cannot make use of the latent word information on availability. To mitigate the shortcomings of both families of approaches, we propose Transformer based Linguistically Informed Sanskrit Tokenizer (TransLIST) consisting of (1) a module that encodes the character input along with latent-word information, which takes into account the sandhi phenomenon specific to SWS and is apt to work with partial or no candidate solutions, (2) a novel soft-masked attention to prioritize potential candidate words and (3) a novel path ranking algorithm to rectify the corrupted predictions. Experiments on the benchmark datasets for SWS show that TransLIST outperforms the current state-of-the-art system by an average 7.2 points absolute gain in terms of perfect match (PM) metric. The codebase and datasets are publicly available at https://github.com/rsingha108/TransLIST