# Tokenization Matters: Improving Zero-Shot NER for Indic Languages

arXiv: https://arxiv.org/abs/2504.16977 · HTML: https://mlsanity.com/paper/2504.16977/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages](https://mlsanity.com/paper/2005.00085/) (2020)
- [Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages](https://mlsanity.com/paper/2212.10168/) (2022)
- [Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy](https://mlsanity.com/paper/2412.17759/) (2024)
- [LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents](https://mlsanity.com/paper/2411.14962/) (2024)
- [Enhancing Document AI Data Generation Through Graph-Based Synthetic Layouts](https://mlsanity.com/paper/2412.03590/) (2024)
- [MVTamperBench: Evaluating Robustness of Vision-Language Models](https://mlsanity.com/paper/2412.19794/) (2024)
- [When Every Token Counts: Optimal Segmentation for Low-Resource Language Models](https://mlsanity.com/paper/2412.06926/) (2024)
- [Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark](https://mlsanity.com/paper/2502.07057/) (2025)
- [NER- RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages](https://mlsanity.com/paper/2412.15252/) (2024)

## Built on by (most cited first, 12 of 12)

- [Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems](https://mlsanity.com/paper/2505.18366/) (2025)
- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding](https://mlsanity.com/paper/2505.17330/) (2025)
- [Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation](https://mlsanity.com/paper/2506.02097/) (2025)
- [IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages](https://mlsanity.com/paper/2602.16832/) (2026)
- [FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Models](https://mlsanity.com/paper/2510.02133/) (2025)
- [LLM-Guided Lifecycle-Aware Clustering of Multi-Turn Customer Support Conversations](https://mlsanity.com/paper/2601.04388/) (2026)
- [Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language](https://mlsanity.com/paper/2510.09032/) (2025)
- [A P=aninian Foundation for Indic Language Processing](https://mlsanity.com/paper/2606.24172/) (2026)
- [BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis](https://mlsanity.com/paper/2607.23319/) (2026)
- [SuTRA : Structurally-Unified Tokenization with Root Awareness](https://mlsanity.com/paper/2608.18087/) (2026)
- [When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages](https://mlsanity.com/paper/2608.27658/) (2026)

## Abstract (arXiv)

Tokenization is a critical component of Natural Language Processing (NLP), especially for low resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character Level tokenization strategies using IndicBERT for NER tasks in low resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties tokenization efficiency, out of vocabulary (OOV) rates, and morphological preservation as well as extrinsic downstream performance, including fine tuning and zero shot cross lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low resource Indic Languages, particularly in zero shot cross lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low resource and morphologically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low resource Indic NLP applications.