mlsanityData to 14 Oct 2026

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

arXiv 2311.09071 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), often show strong performance on English tasks, while exhibiting limitations on other languages. What is an LLM's multilingual capability when it is trained only on certain languages? The underlying mechanism remains unclear. This study endeavors to examine the multilingual capability of LLMs from the vocabulary sharing perspective by conducting an exhaustive analysis across 101 languages. Through the investigation of the performance gap before and after embedding fine-tuning, we discovered four distinct quadrants. By delving into each quadrant we provide actionable and efficient guidelines for tuning these languages. Extensive experiments reveal that existing LLMs possess multilingual capabilities that surpass our expectations, and we can significantly improve the multilingual performance of LLMs based on these attributes of each quadrant~footnoteurlhttps://github.com/CONE-MT/Vocabulary-Sharing-Facilitates-Multilingualism..

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

38 of 47 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
26 more
  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2019How multilingual is Multilingual BERT?cited by 645
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  5. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  6. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  7. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  10. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  11. 2019Massively Multilingual Neural Machine Translationcited by 246
  12. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  13. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  14. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  15. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  16. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  17. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  18. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  19. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  20. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  21. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  22. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  23. 2019How Language-Neutral is Multilingual BERT?cited by 51
  24. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  25. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  26. 2023Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  2. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  5. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 4
  6. 2024Mufu: Multilingual Fused Learning for Low-Resource Translation with LLMcited by 3
  7. 2025Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processingcited by 3
  8. 2025Incorporating Domain Knowledge into Materials Tokenizationcited by 2
  9. 2025EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translationcited by 2
  10. 2026BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languagescited by 2
  11. 2024Evaluation Methodology for Large Language Models for Multilingual Document Question and Answercited by 0
  12. 2024Can linguists better understand DNA?cited by 0