mlsanityData to 7 Oct 2026

A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

arXiv 2404.00929 ↗First posted 1 Apr 2024cs.CL cs.AIcited by 82

Not on the map. Why?

Abstract (arXiv)

Based on the foundation of Large Language Models (LLMs), Multilingual LLMs (MLLMs) have been developed to address the challenges faced in multilingual natural language processing, hoping to achieve knowledge transfer from high-resource languages to low-resource languages. However, significant limitations and challenges still exist, such as language imbalance, multilingual alignment, and inherent bias. In this paper, we aim to provide a comprehensive analysis of MLLMs, delving deeply into discussions surrounding these critical issues. First of all, we start by presenting an overview of MLLMs, covering their evolutions, key techniques, and multilingual capacities. Secondly, we explore the multilingual training corpora of MLLMs and the multilingual datasets oriented for downstream tasks that are crucial to enhance the cross-lingual capability of MLLMs. Thirdly, we survey the state-of-the-art studies of multilingual representations and investigate whether the current MLLMs can learn a universal language representation. Fourthly, we discuss bias on MLLMs, including its categories, evaluation metrics, and debiasing techniques. Finally, we discuss existing challenges and point out promising research directions of MLLMs.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 145 of its 200 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

145 of 200 references.

References · 145

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  13. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  14. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  15. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  16. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  19. 2023PaLM 2 Technical Reportcited by 1,154
  20. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  21. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  22. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  23. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  24. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  25. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  26. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  27. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  28. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  29. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  30. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  31. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  32. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  33. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  34. 2019How multilingual is Multilingual BERT?cited by 642
  35. 2023A Comprehensive Overview of Large Language Modelscited by 639
  36. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  37. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  38. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518

50 most cited of 145 All 145 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 37; 2026: 24.

82 papers.

Sort:

By area:

  1. 2024Large Language Models Meet NLP: A Surveycited by 102
  2. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  3. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  4. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  5. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  6. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  7. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  8. 2024Linguini: A benchmark for language-agnostic linguistic reasoningcited by 17
  9. 2025Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMscited by 15
  10. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  11. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  12. 2024L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Contextcited by 10
38 more
  1. 2024Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICscited by 10
  2. 2025CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answeringcited by 10
  3. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 10
  4. 20241+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?cited by 9
  5. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  6. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  7. 2025HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMscited by 8
  8. 2025Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classificationcited by 7
  9. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  10. 2024Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generationcited by 6
  11. 2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
  12. 2025Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMscited by 6
  13. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  14. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  15. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  16. 2024Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performancecited by 4
  17. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4
  18. 2025One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoningcited by 4
  19. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 4
  20. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impactcited by 3
  21. 2024Make Satire Boring Again: Reducing Stylistic Bias of Satirical Corpus by Utilizing Generative LLMscited by 3
  22. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  23. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 3
  24. 2024FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Modelscited by 2
  25. 2024Towards Safe Multilingual Frontier AIcited by 2
  26. 2025Translation in the Wildcited by 2
  27. 2025Assessing the Role of Data Quality in Training Bilingual Language Modelscited by 2
  28. 2025KROMA: Ontology Matching with Knowledge Retrieval and Large Language Modelscited by 2
  29. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  30. 2025Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Modelscited by 2
  31. 2026A Scalable Entity-Based Framework for Auditing Bias in Large Language Modelscited by 2
  32. 2026ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Lawscited by 2
  33. 2024sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Promptingcited by 1
  34. 2024A Comparative Study of Translation Bias and Accuracy in Multilingual Large Language Models for Cross-Language Claim Verificationcited by 1
  35. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  36. 2025How much does context affect the accuracy of AI health advice?cited by 1
  37. 2025Evaluating Visual Mathematics in Multimodal LLMs: A Multilingual Benchmark Based on the Kangaroo Testscited by 1
  38. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundariescited by 1

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  2. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
  3. 2024Concept Space Alignment in Multilingual LLMs
  4. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
  5. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  6. 2024What Drives Performance in Multilingual Language Models?
  7. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data
  8. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages