mlsanityData to 14 Oct 2026

Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders

arXiv 2511.10840 ↗First posted 13 Nov 2025cs.CLcited by 8

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does performance favor the dominant training language? To address this, we train models on different multilingual mixtures and analyze their internal mechanisms using Cross-Layer Transcoders (CLTs) and Attribution Graphs. Our results reveal multilingual shared representations: the model employs highly similar features across languages, while language-specific decoding emerges in later layers. Training models without English shows identical multilingual shared space structures. Decoding relies partly on a small set of high-frequency features in the final layers, which linearly encode language identity from early layers. Intervening on these features allows one language to be suppressed and another substituted. Finally, to explain non-English failures, we perform a Model-Diffing experiment: underperformance arises from dim late-layer features, weak middle-layer clusters, and tokenizer bias toward English that forces early layers to specialize in word reassembly. Finetuning strengthens these features and their links, improving token assembly and language-specific decoding, providing a mechanistic explanation for multilingual gaps. Our models and CLTs are available at https://huggingface.co/collections/CausalNLP/multilingual-clts and https://huggingface.co/collections/CausalNLP/multilingual-gpt2-models. Our code is available at: https://github.com/abirharrasse/MultilingualCLTs

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 21 of its 33 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

21 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Do Multilingual LLMs Think In English?21 Feb
  2. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs11 Feb

Other references · 19

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  8. 2019How multilingual is Multilingual BERT?cited by 645
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  10. 2024The Platonic Representation Hypothesiscited by 382
  11. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  12. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
7 more
  1. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  2. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  3. 2023Do Multilingual Language Models Think Better in English?cited by 88
  4. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  5. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  6. 2020From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformerscited by 35
  7. 2025Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025Intrinsically Interpretable Attention via Sparse Post-Trainingcited by 4
  2. 2026CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphscited by 2
  3. 2026Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representationscited by 0
  4. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?cited by 0
  5. 2026How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attributioncited by 0
  6. 2026Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Modelscited by 0
  7. 2026Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMscited by 0
  8. 2026A Systematic Comparison of Multilingual Interpretability Methods Reveals Anisotropy-Driven Failurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the role of FFNs in driving multilingual behaviour in LLMs
  2. 2025How Language Directions Align with Token Geometry in Multilingual LLMs
  3. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  4. 2026The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space
  5. 2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders
  6. 2025Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons
  7. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling
  8. 2024Probing the Emergence of Cross-lingual Alignment during LLM Training