# Do Multilingual LLMs Think In English?

arXiv: https://arxiv.org/abs/2502.15603 · HTML: https://mlsanity.com/paper/2502.15603/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 34)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [Future Lens: Anticipating Subsequent Tokens from a Single Hidden State](https://mlsanity.com/paper/2311.04897/) (2023)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [Is Cosine-Similarity of Embeddings Really About Similarity?](https://mlsanity.com/paper/2403.05440/) (2024)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [Extrapolating Large Language Models to Non-English by Aligning Languages](https://mlsanity.com/paper/2308.04948/) (2023)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.07037/) (2024)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [Who's asking? User personas and the mechanics of latent misalignment](https://mlsanity.com/paper/2406.12094/) (2024)

## Built on by (most cited first, 30 of 78)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Refusal Direction is Universal Across Safety-Aligned Languages](https://mlsanity.com/paper/2505.17306/) (2025)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [The Dual-Route Model of Induction](https://mlsanity.com/paper/2504.03022/) (2025)
- [Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning](https://mlsanity.com/paper/2507.05418/) (2025)
- [R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning](https://mlsanity.com/paper/2502.19735/) (2025)
- [Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline](https://mlsanity.com/paper/2505.20546/) (2025)
- [The Emergence of Abstract Thought in Large Language Models Beyond Any Language](https://mlsanity.com/paper/2506.09890/) (2025)
- [Localized Cultural Knowledge is Conserved and Controllable in Large Language Models](https://mlsanity.com/paper/2504.10191/) (2025)
- [Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?](https://mlsanity.com/paper/2505.17407/) (2025)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding](https://mlsanity.com/paper/2509.23050/) (2025)
- [Beyond Transcription: Mechanistic Interpretability in ASR](https://mlsanity.com/paper/2508.15882/) (2025)
- [Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models](https://mlsanity.com/paper/2506.05850/) (2025)
- [The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure](https://mlsanity.com/paper/2506.22724/) (2025)
- [Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering](https://mlsanity.com/paper/2601.14050/) (2026)
- [CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation](https://mlsanity.com/paper/2507.14239/) (2025)
- [Activation Steering for Masked Diffusion Language Models](https://mlsanity.com/paper/2512.24143/) (2025)
- [Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders](https://mlsanity.com/paper/2511.10840/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla](https://mlsanity.com/paper/2509.09101/) (2025)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025)
- [When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training](https://mlsanity.com/paper/2601.22851/) (2026)
- [Language Specific Knowledge: Do Models Know Better in X than in English?](https://mlsanity.com/paper/2505.14990/) (2025)
- [Under the Shadow of Babel: How Language Shapes Reasoning in LLMs](https://mlsanity.com/paper/2506.16151/) (2025)
- [Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs](https://mlsanity.com/paper/2509.17701/) (2025)
- [On the Entity-Level Alignment in Crosslingual Consistency](https://mlsanity.com/paper/2510.10280/) (2025)
- [Text2Cypher Across Languages: Evaluating and Finetuning LLMs](https://mlsanity.com/paper/2506.21445/) (2025)
- [Humans overrely on overconfident language models, across languages](https://mlsanity.com/paper/2507.06306/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have multilingual capabilities and can solve tasks across various languages. However, we show that current LLMs make key decisions in a representation space closest to English, regardless of their input and output languages. Exploring the internal representations with a logit lens for sentences in French, German, Dutch, and Mandarin, we show that the LLM first emits representations close to English for semantically-loaded words before translating them into the target language. We further show that activation steering in these LLMs is more effective when the steering vectors are computed in English rather than in the language of the inputs and outputs. This suggests that multilingual LLMs perform key reasoning steps in a representation that is heavily shaped by English in a way that is not transparent to system users.