mlsanityData to 7 Oct 2026

MaLA-500: Massive Language Adaptation of Large Language Models

arXiv 2401.13303 ↗First posted 24 Jan 2024cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs. Moreover, the extrinsic evaluation of in-context learning shows that MaLA-500 outperforms previous LLMs on SIB200 and Taxi1500 by a significant margin, i.e., 11.68% and 4.82% marco-average accuracy across languages. We release MaLA-500 at https://huggingface.co/MaLA-LM

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

45 of 57 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
33 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  3. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  4. 2022UL2: Unifying Language Learning Paradigmscited by 292
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  6. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  7. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  8. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  9. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  10. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  11. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  12. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  13. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  14. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  15. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  16. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  17. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  18. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  19. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  20. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  21. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  22. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  23. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
  24. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  25. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  26. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 48
  27. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  28. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  29. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  30. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  31. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  32. 2020Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizicited by 13
  33. 2020Parsing with Multilingual BERT, a Small Corpus, and a Small Treebankcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 10; 2026: 4.

28 papers.

Sort:

By area:

  1. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  2. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  3. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  4. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  5. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  6. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  7. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  8. 2024Recovering the Pre-Fine-Tuning Weights of Generative Modelscited by 15
  9. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  10. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  11. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  12. 2025SinLlama -- A Large Language Model for Sinhalacited by 8
16 more
  1. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  2. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  3. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  4. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  5. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  6. 2024Data-Augmentation-Based Dialectal Adaptation for LLMscited by 3
  7. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languagescited by 3
  8. 2025Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuningcited by 3
  9. 2024Improving Multilingual ASR in the Wild Using Simple N-best Re-rankingcited by 2
  10. 2025Breaking Language Barriers: Equitable Performance in Multilingual Language Modelscited by 2
  11. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  12. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  13. 2026CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMscited by 0
  14. 2026Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settingscited by 0
  15. 2026Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malaycited by 0
  16. 2026In-Place Tokenizer Expansion for Pre-trained LLMscited by 0