mlsanityData to 7 Oct 2026

On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons

arXiv 2404.02431 ↗First posted 3 Apr 2024cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

Current decoder-based pre-trained language models (PLMs) successfully demonstrate multilingual capabilities. However, it is unclear how these models handle multilingualism. We analyze the neuron-level internal behavior of multilingual decoder-based PLMs, Specifically examining the existence of neurons that fire ``uniquely for each language'' within decoder-only multilingual PLMs. We analyze six languages: English, German, French, Spanish, Chinese, and Japanese, and show that language-specific neurons are unique, with a slight overlap (< 5%) between languages. These neurons are mainly distributed in the models' first and last few layers. This trend remains consistent across languages and models. Additionally, we tamper with less than 1% of the total neurons in each model during inference and demonstrate that tampering with a few language-specific neurons drastically changes the probability of target language occurrence in text generation.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 10 of its 33 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

10 of 33 references.

References · 10

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  6. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  7. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  8. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  9. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  10. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 38; 2026: 28.

83 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  4. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  5. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  6. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  7. 2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
  8. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  9. 2024Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMscited by 20
  10. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  11. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  12. 2024Multilingual Knowledge Editing with Language-Agnostic Factual Neuronscited by 18
38 more
  1. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  2. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  3. 2025Disentangling Language and Culture for Evaluating Multilingual Large Language Modelscited by 15
  4. 2024Knowledge Localization: Mission Not Accomplished? Enter Query Localization!cited by 13
  5. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoderscited by 12
  6. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMscited by 11
  7. 2025Language-specific Neurons Do Not Facilitate Cross-Lingual Transfercited by 11
  8. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 11
  9. 2024Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passescited by 10
  10. 2025Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Modelscited by 10
  11. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 9
  12. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 9
  13. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
  14. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 9
  15. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  16. 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
  17. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 8
  18. 2024MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Modelscited by 7
  19. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  20. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  21. 2024Lens: Rethinking Multilingual Enhancement for Large Language Modelscited by 5
  22. 2025How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoderscited by 5
  23. 2025Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?cited by 5
  24. 2025ReCoVeR the Target Language: Language Steering without Sacrificing Task Performancecited by 5
  25. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  26. 2026When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Trainingcited by 5
  27. 2025On Relation-Specific Neurons in Large Language Modelscited by 4
  28. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 4
  29. 2025What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representationscited by 4
  30. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  31. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Modelscited by 4
  32. 2026Cross-Lingual Activation Steering for Multilingual Language Modelscited by 4
  33. 2025Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learningcited by 3
  34. 2025Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMscited by 3
  35. 2025Unveiling the Influence of Amplifying Language-Specific Neuronscited by 3
  36. 2025When Alignment Hurts: Decoupling Representational Spaces in Multilingual Modelscited by 3
  37. 2024The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Modelcited by 2
  38. 2025M-Wanda: Improving One-Shot Pruning for Multilingual LLMscited by 2

50 most cited of 83 All 83 (JSON)