# Language-Specific Latent Process Hinders Cross-Lingual Performance

arXiv: https://arxiv.org/abs/2505.13141 · HTML: https://mlsanity.com/paper/2505.13141/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 53 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 43)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization](https://mlsanity.com/paper/2401.06838/) (2024)
- [The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities](https://mlsanity.com/paper/2411.04986/) (2024)
- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models](https://mlsanity.com/paper/2504.04264/) (2025)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [Question Translation Training for Better Multilingual Reasoning](https://mlsanity.com/paper/2401.07817/) (2024)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)

## Built on by (most cited first, 21 of 21)

- [Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline](https://mlsanity.com/paper/2505.20546/) (2025)
- [Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment](https://mlsanity.com/paper/2602.16660/) (2026)
- [Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs](https://mlsanity.com/paper/2510.26024/) (2025)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [Cross-Lingual Activation Steering for Multilingual Language Models](https://mlsanity.com/paper/2601.16390/) (2026)
- [On the Entity-Level Alignment in Crosslingual Consistency](https://mlsanity.com/paper/2510.10280/) (2025)
- [Identity-Aware Large Language Models require Cultural Reasoning](https://mlsanity.com/paper/2510.18510/) (2025)
- [Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness](https://mlsanity.com/paper/2510.06780/) (2025)
- [Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts](https://mlsanity.com/paper/2603.17070/) (2026)
- [Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility](https://mlsanity.com/paper/2606.03291/) (2026)
- [Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops](https://mlsanity.com/paper/2608.28860/) (2026)
- [Code-Switching Reveals Language Anchoring in Multilingual LLMs](https://mlsanity.com/paper/2606.19668/) (2026)
- [The Illusion of Cross-Lingual Safety in Low-Resource Languages](https://mlsanity.com/paper/2608.11146/) (2026)
- [Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations](https://mlsanity.com/paper/2605.01609/) (2026)
- [Speculative Decoding and the Curse of Multilinguality](https://mlsanity.com/paper/2605.30580/) (2026)
- [Soft Token Alignment for Cross-Lingual Reasoning](https://mlsanity.com/paper/2606.26466/) (2026)
- [Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages](https://mlsanity.com/paper/2608.00533/) (2026)
- [Choosing the Right Language Mode at Inference Time for Multilingual Reliability](https://mlsanity.com/paper/2609.04653/) (2026)
- [Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space](https://mlsanity.com/paper/2609.06381/) (2026)
- [Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning](https://mlsanity.com/paper/2609.10445/) (2026)
- [Retrieval Disagreement as a Signal: Diagnosing and Steering Multilingual Dense Embedders](https://mlsanity.com/paper/2610.18526/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are demonstrably capable of cross-lingual transfer, but can produce inconsistent output when prompted with the same queries written in different languages. To understand how language models are able to generalize knowledge from one language to the others, we measure representation similarity between languages, and apply the logit lens to interpret the implicit steps taken by LLMs to solve multilingual multi-choice reasoning questions. Our analyses reveal LLMs predict inconsistently and are less accurate because they rely on representations that are dissimilar across languages, rather than working in a shared semantic space. While larger models are more multilingual, we show their hidden states are more likely to dissociate from the shared representation compared to smaller models, but are nevertheless more capable of retrieving knowledge embedded across different languages. Finally, we demonstrate that knowledge sharing in small models can be facilitated by steering their latent processing towards the shared semantic space. This improves the models' multilingual reasoning performance, as a result of more knowledge transfer from, and better output consistency with English.