Translation Heads: Disentangling meaning from language in LLM-based machine translation
On the map: Mechanistic interpretability → Knowledge and representations inside LLMs
Abstract (arXiv)
Mechanistic Interpretability (MI) seeks to explain how neural networks implement their capabilities, but the scale of Large Language Models (LLMs) has limited prior MI work in Machine Translation (MT) to word-level analyses. We study sentence-level MT from a mechanistic perspective by analyzing attention heads to understand how LLMs internally encode and distribute translation functions. We decompose MT into two subtasks: producing text in the target language (i.e. target language identification) and preserving the input sentence's meaning (i.e. sentence equivalence). Across three families of open-source models and 20 translation directions, we find that distinct, sparse sets of attention heads specialize in each subtask. Based on this insight, we construct subtask-specific steering vectors and show that modifying just 1% of the relevant heads enables instruction-free MT performance comparable to instruction-based prompting, while ablating these heads selectively disrupts their corresponding translation functions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. First posted on arXiv 4 Feb 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space
- 2024Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference
- 2026Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs
- 2023Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models
- 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective
- 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models
- 2024Rethinking Interpretability in the Era of Large Language Models
- 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation