mlsanityData to 7 Oct 2026

Do Llamas Work in English? On the Latent Language of Multilingual Transformers

arXiv 2402.10588 ↗First posted 16 Feb 2024cs.CL cs.CYcited by 275

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output embedding from which next-token probabilities are computed. Tracking intermediate embeddings through their high-dimensional space reveals three distinct phases, whereby intermediate embeddings (1) start far away from output token embeddings; (2) already allow for decoding a semantically correct next token in the middle layers, but give higher probability to its version in English than in the input language; (3) finally move into an input-language-specific region of the embedding space. We cast these results into a conceptual model where the three phases operate in "input space", "concept space", and "output space", respectively. Crucially, our evidence suggests that the abstract "concept space" lies closer to English than to other languages, which may have important consequences regarding the biases held by multilingual language models.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2014How transferable are features in deep neural networks?cited by 1,216
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
29 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  7. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  8. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  9. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  12. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  13. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  14. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  15. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  16. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  17. 2023Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMscited by 103
  18. 2023Do Multilingual Language Models Think Better in English?cited by 87
  19. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  20. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
  21. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  22. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  23. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  24. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  25. 2024Question Translation Training for Better Multilingual Reasoningcited by 41
  26. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  27. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  28. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  29. 2023Training Dynamics of Contextual N-Grams in Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 106; 2026: 112.

275 papers.

Sort:

By area:

  1. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  3. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  4. 2025Do Multilingual LLMs Think In English?cited by 75
  5. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  6. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  7. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  8. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  9. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  10. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  11. 2024Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?cited by 40
  12. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
38 more
  1. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  2. 2024AI and the Problem of Knowledge Collapsecited by 34
  3. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  4. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  5. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
  6. 2025Large Language Models Are Human-Like Internallycited by 28
  7. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  8. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  9. 2024Data Contamination Can Cross Language Barrierscited by 26
  10. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  11. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  12. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  13. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  14. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 22
  15. 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMscited by 22
  16. 2025LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Modelscited by 22
  17. 2025The Dual-Route Model of Inductioncited by 22
  18. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  19. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  20. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  21. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  22. 2025Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causescited by 20
  23. 2025Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipelinecited by 20
  24. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  25. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  26. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  27. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  28. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  29. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  30. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  31. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignmentcited by 17
  32. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  33. 2025SEA-LION: Southeast Asian Languages in One Networkcited by 17
  34. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  35. 2024Large Language Models Are Cross-Lingual Knowledge-Free Reasonerscited by 16
  36. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  37. 2025Disentangling Language and Culture for Evaluating Multilingual Large Language Modelscited by 15
  38. 2025AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thoughtcited by 14

50 most cited of 275 All 275 (JSON)