mlsanityData to 7 Oct 2026

How do Large Language Models Handle Multilingualism?

arXiv 2402.18815 ↗First posted 29 Feb 2024cs.CL cs.AIcited by 148

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow (textttMWork): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. To verify textttMWork, we introduce Parallel Language-specific Neuron Detection (textttPLND) to identify activated neurons for inputs in different languages without any labeled data. Using textttPLND, we validate textttMWork through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. Moreover, textttMWork allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of 3.6% for high-resource languages and 2.3% for low-resource languages across all tasks with just 400 documents.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

37 of 50 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
25 more
  1. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  2. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  4. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  5. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  6. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  7. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  8. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  9. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  10. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  11. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  12. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  13. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  14. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  15. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  16. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  17. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  18. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  19. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  20. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  21. 2022Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structurecited by 17
  22. 2023Enhancing Cross-lingual Transfer via Phonemic Transcription Integrationcited by 11
  23. 2021On the Language-specificity of Multilingual BERT and the Impact of Fine-tuningcited by 8
  24. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  25. 2023Machine-Created Universal Language for Cross-lingual Transfercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 74; 2026: 41.

148 papers.

Sort:

By area:

  1. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  2. 2024Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectorscited by 61
  3. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  4. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  5. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  6. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 35
  7. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  8. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  9. 2024Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Modelscited by 31
  10. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  11. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 28
  12. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
38 more
  1. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  2. 2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
  3. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 23
  4. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  5. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  6. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  7. 2024Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMscited by 20
  8. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  9. 2025Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipelinecited by 20
  10. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  11. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  12. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  13. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignmentcited by 17
  14. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  15. 2024Large Language Models Are Cross-Lingual Knowledge-Free Reasonerscited by 16
  16. 2025Disentangling Language and Culture for Evaluating Multilingual Large Language Modelscited by 15
  17. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14
  18. 2024Knowledge Localization: Mission Not Accomplished? Enter Query Localization!cited by 13
  19. 2024Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialectcited by 13
  20. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  21. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoderscited by 12
  22. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learnerscited by 11
  23. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMscited by 11
  24. 2025Language-specific Neurons Do Not Facilitate Cross-Lingual Transfercited by 11
  25. 2025Improving Multilingual Language Models by Aligning Representations through Steeringcited by 11
  26. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 11
  27. 2024Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passescited by 10
  28. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  29. 2025Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metriccited by 10
  30. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 10
  31. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  32. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  33. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 9
  34. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
  35. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  36. 2025PerCul: A Story-Driven Cultural Evaluation of LLMs in Persiancited by 8
  37. 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
  38. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8

50 most cited of 148 All 148 (JSON)