mlsanityData to 14 Oct 2026

Accelerating Multilingual Language Model for Excessively Tokenized Languages

arXiv 2401.10660 ↗First posted 19 Jan 2024cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have remarkably enhanced performances on a variety of tasks in multiple languages. However, tokenizers in LLMs trained primarily on English-centric corpora often overly fragment a text into character or Unicode-level tokens in non-Roman alphabetic languages, leading to inefficient text generation. We introduce a simple yet effective framework to accelerate text generation in such languages. Our approach involves employing a new language model head with a vocabulary set tailored to a specific target language for a pre-trained LLM. This is followed by fine-tuning the new head while incorporating a verification step to ensure the model's performance is preserved. We show that this targeted fine-tuning, while freezing other model parameters, effectively reduces token fragmentation for the target language. Our extensive experiments demonstrate that the proposed framework increases the generation speed by a factor of 1.7 while maintaining the performance of pre-trained multilingual models on target monolingual tasks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

38 of 56 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2020GLU Variants Improve Transformercited by 1,804
26 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2018Hierarchical Neural Story Generationcited by 1,193
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  10. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  12. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  13. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  14. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  15. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  16. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  17. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  18. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  19. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  20. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  21. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 28
  22. 2020Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfercited by 23
  23. 2021AVocaDo: Strategy for Adapting Vocabulary to Downstream Domaincited by 20
  24. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  25. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  26. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 3.

8 papers.

  1. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  2. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  3. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  4. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  5. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  6. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  7. 2026Modular Monolingual Adaptation using Pretrained Language Modelscited by 0
  8. 2026Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinyacited by 0