# Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs

arXiv: https://arxiv.org/abs/2609.01356 · HTML: https://mlsanity.com/paper/2609.01356/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://mlsanity.com/paper/2303.07992/) (2023)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [pyvene: A Library for Understanding and Improving PyTorch Models via Interventions](https://mlsanity.com/paper/2403.07809/) (2024)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources](https://mlsanity.com/paper/2211.15649/) (2022)
- [Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation](https://mlsanity.com/paper/2404.11201/) (2024)
- [Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation](https://mlsanity.com/paper/2507.22608/) (2025)
- [LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encoder](https://mlsanity.com/paper/2502.20344/) (2025)
- [CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark](https://mlsanity.com/paper/2601.08331/) (2026)
- [Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders](https://mlsanity.com/paper/2511.10840/) (2025)

## Abstract (arXiv)

Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.