# Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?

arXiv: https://arxiv.org/abs/2408.10811 · HTML: https://mlsanity.com/paper/2408.10811/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs](https://mlsanity.com/paper/2407.03963/) (2024)
- [SambaLingo: Teaching Large Language Models New Languages](https://mlsanity.com/paper/2404.05829/) (2024)
- [The Claire French Dialogue Dataset](https://mlsanity.com/paper/2311.16840/) (2023)

## Built on by (most cited first, 30 of 41)

- [MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation](https://mlsanity.com/paper/2503.10497/) (2025)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment](https://mlsanity.com/paper/2410.05873/) (2024)
- [SEA-LION: Southeast Asian Languages in One Network](https://mlsanity.com/paper/2504.05747/) (2025)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [The Emergence of Abstract Thought in Large Language Models Beyond Any Language](https://mlsanity.com/paper/2506.09890/) (2025)
- [AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thought](https://mlsanity.com/paper/2501.16154/) (2025)
- [Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2505.05111/) (2025)
- [RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs](https://mlsanity.com/paper/2502.07424/) (2025)
- [The Impact of Language Mixing on Bilingual LLM Reasoning](https://mlsanity.com/paper/2507.15849/) (2025)
- [Improving Multilingual Language Models by Aligning Representations through Steering](https://mlsanity.com/paper/2505.12584/) (2025)
- [Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models](https://mlsanity.com/paper/2506.05850/) (2025)
- [Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models](https://mlsanity.com/paper/2505.16538/) (2025)
- [Benchmark of stylistic variation in LLM-generated texts](https://mlsanity.com/paper/2509.10179/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level](https://mlsanity.com/paper/2604.02178/) (2026)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models](https://mlsanity.com/paper/2412.12500/) (2024)
- [AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment](https://mlsanity.com/paper/2509.24338/) (2025)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations](https://mlsanity.com/paper/2507.20279/) (2025)
- [Language Specific Knowledge: Do Models Know Better in X than in English?](https://mlsanity.com/paper/2505.14990/) (2025)
- [Text2Cypher Across Languages: Evaluating and Finetuning LLMs](https://mlsanity.com/paper/2506.21445/) (2025)
- [From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Task](https://mlsanity.com/paper/2508.12774/) (2025)
- [The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model](https://mlsanity.com/paper/2412.07298/) (2024)
- [The Paradox of Poetic Intent in Back-Translation: Evaluating the Quality of Large Language Models in Chinese Translation](https://mlsanity.com/paper/2504.16286/) (2025)
- [Breaking Language Barriers: Equitable Performance in Multilingual Language Models](https://mlsanity.com/paper/2508.12662/) (2025)
- [Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models](https://mlsanity.com/paper/2511.07498/) (2025)

## Abstract (arXiv)

In this study, we investigate whether non-English-centric LLMs, despite their strong performance, `think' in their respective dominant language: more precisely, `think' refers to how the representations of intermediate layers, when un-embedded into the vocabulary space, exhibit higher probabilities for certain dominant languages during generation. We term such languages as internal latent languages. We examine the latent language of three typical categories of models for Japanese processing: Llama2, an English-centric model; Swallow, an English-centric model with continued pre-training in Japanese; and LLM-jp, a model pre-trained on balanced English and Japanese corpora. Our empirical findings reveal that, unlike Llama2 which relies exclusively on English as the internal latent language, Japanese-specific Swallow and LLM-jp employ both Japanese and English, exhibiting dual internal latent languages. For any given target language, the model preferentially activates the latent language most closely related to it. In addition, we explore how intermediate layers respond to questions involving cultural conflicts between latent internal and target output languages. We further explore how the language identity shifts across layers while keeping consistent semantic meaning reflected in the intermediate layer representations. This study deepens the understanding of non-English-centric large language models, highlighting the intricate dynamics of language representation within their intermediate layers.