# Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

arXiv: https://arxiv.org/abs/2410.03718 · HTML: https://mlsanity.com/paper/2410.03718/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [What Makes for Good Visual Tokenizers for Large Language Models?](https://mlsanity.com/paper/2305.12223/) (2023)
- [Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?](https://mlsanity.com/paper/2407.16607/) (2024)
- [SUTRA: Scalable Multilingual Language Model Architecture](https://mlsanity.com/paper/2405.06694/) (2024)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://mlsanity.com/paper/2403.11130/) (2024)
- [AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese](https://mlsanity.com/paper/2305.13641/) (2023)
- [AsPOS: Assamese Part of Speech Tagger using Deep Learning Approach](https://mlsanity.com/paper/2212.07043/) (2022)

## Built on by (most cited first, 3 of 3)

- [Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages](https://mlsanity.com/paper/2411.12240/) (2024)
- [Enforcement Agents: Enhancing Accountability and Resilience in Multi-Agent AI Frameworks](https://mlsanity.com/paper/2504.04070/) (2025)
- [An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settings](https://mlsanity.com/paper/2410.11866/) (2024)

## Abstract (arXiv)

Training of a tokenizer plays an important role in the performance of deep learning models. This research aims to understand the performance of tokenizers in five state-of-the-art (SOTA) large language models (LLMs) in the Assamese language of India. The research is important to understand the multi-lingual support for a low-resourced language such as Assamese. Our research reveals that the tokenizer of SUTRA from Two AI performs the best with an average Normalized Sequence Length (NSL) value of 0.45, closely followed by the tokenizer of GPT-4o from Open AI with an average NSL value of 0.54, followed by Gemma 2, Meta Llama 3.1, and Mistral Large Instruct 2407 with an average NSL value of 0.82, 1.4, and 1.48 respectively.