mlsanityData to 14 Oct 2026

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

arXiv 2410.03718 ↗First posted 28 Sep 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Training of a tokenizer plays an important role in the performance of deep learning models. This research aims to understand the performance of tokenizers in five state-of-the-art (SOTA) large language models (LLMs) in the Assamese language of India. The research is important to understand the multi-lingual support for a low-resourced language such as Assamese. Our research reveals that the tokenizer of SUTRA from Two AI performs the best with an average Normalized Sequence Length (NSL) value of 0.45, closely followed by the tokenizer of GPT-4o from Open AI with an average NSL value of 0.54, followed by Gemma 2, Meta Llama 3.1, and Mistral Large Instruct 2407 with an average NSL value of 0.82, 1.4, and 1.48 respectively.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 14 of its 36 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

14 of 36 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  5. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  6. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  7. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  8. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  9. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  10. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  11. 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
  12. 2024Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Modelscited by 5
2 more
  1. 2023AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamesecited by 4
  2. 2022AsPOS: Assamese Part of Speech Tagger using Deep Learning Approachcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 11
  2. 2025Enforcement Agents: Enhancing Accountability and Resilience in Multi-Agent AI Frameworkscited by 4
  3. 2024An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settingscited by 0