# One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks

arXiv: https://arxiv.org/abs/2409.13920 · HTML: https://mlsanity.com/paper/2409.13920/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages](https://mlsanity.com/paper/2403.06350/) (2024)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?](https://mlsanity.com/paper/2005.01330/) (2020)
- [TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer](https://mlsanity.com/paper/2210.11753/) (2022)
- [A Benchmark and Dataset for Post-OCR text correction in Sanskrit](https://mlsanity.com/paper/2211.07980/) (2022)
- [Evaluating Neural Morphological Taggers for Sanskrit](https://mlsanity.com/paper/2005.10893/) (2020)

## Built on by (most cited first, 15 of 15)

- [MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for P=ali, Sanskrit, Buddhist Chinese, and Tibetan](https://mlsanity.com/paper/2601.06400/) (2026)
- [BanglaByT5: Byte-Level Modelling for Bangla](https://mlsanity.com/paper/2505.17102/) (2025)
- [Still Not There: Can LLMs Outperform Smaller Task-Specific Seq2Seq Models on the Poetry-to-Prose Conversion Task?](https://mlsanity.com/paper/2511.08145/) (2025)
- [Uni-Hema: Unified Model for Digital Hematopathology](https://mlsanity.com/paper/2511.13889/) (2025)
- [PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks](https://mlsanity.com/paper/2605.16881/) (2026)
- [Mapping Hymns and Organizing Concepts in the Rigveda: Quantitatively Connecting the Vedic Suktas](https://mlsanity.com/paper/2503.18226/) (2025)
- [A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMs](https://mlsanity.com/paper/2505.13173/) (2025)
- [Accent Placement Models for Rigvedic Sanskrit Text](https://mlsanity.com/paper/2511.23088/) (2025)
- [From Outliers to Errors: Auditing Pali-to-English LLM Translations with Multi-Reference Adjudication](https://mlsanity.com/paper/2606.01136/) (2026)
- [A P=aninian Foundation for Indic Language Processing](https://mlsanity.com/paper/2606.24172/) (2026)
- [BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis](https://mlsanity.com/paper/2607.23319/) (2026)
- [Pad=artha: Ontology-Grounded Fine-Grained NER Benchmark for Classical Sanskrit](https://mlsanity.com/paper/2608.29324/) (2026)
- [Vectorizing Classical Tamil: Representation Learning for Verse-Commentary Pairs](https://mlsanity.com/paper/2609.04755/) (2026)
- [Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition](https://mlsanity.com/paper/2609.12960/) (2026)
- [A Computational Approach to Measuring Semantic Change in Sanskrit Literature](https://mlsanity.com/paper/2609.25012/) (2026)

## Abstract (arXiv)

Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morphologically rich language Sanskrit. We evaluate ByT5-Sanskrit on established Sanskrit word segmentation tasks, where it outperforms previous data-driven approaches by a considerable margin and matches the performance of the current best lexicon-based model. It is easier to deploy and more robust to data not covered by external linguistic resources. It also achieves new state-of-the-art results in Vedic Sanskrit dependency parsing and OCR post-correction tasks. Additionally, based on the Digital Corpus of Sanskrit, we introduce a novel multitask dataset for the joint training of Sanskrit word segmentation, lemmatization, and morphosyntactic tagging tasks. We fine-tune ByT5-Sanskrit on this dataset, creating a versatile multitask model for various downstream Sanskrit applications. We have used this model in Sanskrit linguistic annotation projects, in information retrieval setups, and as a preprocessing step in a Sanskrit machine translation pipeline. We also show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages. We thus demonstrate that byte-level pretrained language models can achieve excellent performance for morphologically rich languages, outperforming tokenizer-based models and presenting an important vector of exploration when constructing NLP pipelines for such languages.