mlsanityData to 14 Oct 2026

Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers

arXiv 2411.08745 ↗First posted 13 Nov 2024cs.CL cs.AIcited by 44

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

A central question in multilingual language modeling is whether large language models (LLMs) develop a universal concept representation, disentangled from specific languages. In this paper, we address this question by analyzing latent representations (latents) during a word-translation task in transformer-based LLMs. We strategically extract latents from a source translation prompt and insert them into the forward pass on a target translation prompt. By doing so, we find that the output language is encoded in the latent at an earlier layer than the concept to be translated. Building on this insight, we conduct two key experiments. First, we demonstrate that we can change the concept without changing the language and vice versa through activation patching alone. Second, we show that patching with the mean representation of a concept across different languages does not affect the models' ability to translate it, but instead improves it. Finally, we generalize to multi-token generation and demonstrate that the model can generate natural language description of those mean representations. Our results provide evidence for the existence of language-agnostic concept representations within the investigated models.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 50 references that are in the map. First posted on arXiv 13 Nov 2024.

Builds on

33 of 50 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023Mistral 7Bcited by 3,164
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  10. 2023Whose Opinions Do Language Models Reflect?cited by 773
  11. 2019How multilingual is Multilingual BERT?cited by 645
  12. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
21 more
  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  2. 2023Function Vectors in Large Language Modelscited by 308
  3. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  5. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
  6. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  7. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  8. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  9. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 85
  10. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  11. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  12. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 67
  13. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 45
  14. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  15. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 23
  16. 2024How Do Multilingual Language Models Remember Facts?cited by 20
  17. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  18. 2023Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Modelscited by 18
  19. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  20. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  21. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 31.

44 papers.

Sort:

By area:

  1. 2025The Dual-Route Model of Inductioncited by 24
  2. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  3. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMscited by 12
  4. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMscited by 12
  5. 2025Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMscited by 9
  6. 2025LLMs Process Lists With General Filter Headscited by 7
  7. 2025Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Modelscited by 6
  8. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  9. 2025What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representationscited by 5
  10. 2026When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Trainingcited by 5
  11. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
  12. 2025On the Entity-Level Alignment in Crosslingual Consistencycited by 4
32 more
  1. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  2. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  3. 2026Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Modelscited by 3
  4. 2026In-Context Learning Operates as Concept Subspace Learningcited by 3
  5. 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Modelscited by 2
  6. 2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptationcited by 2
  7. 2026Layer-Targeted Multilingual Knowledge Erasure in Large Language Modelscited by 2
  8. 2026How LLMs Are Persuaded: A Few Attention Heads, Reroutedcited by 2
  9. 2026Low-Resource Safety Failures Are Action Failures, Not Representation Failurescited by 2
  10. 2026Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generationcited by 1
  11. 2026LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steeringcited by 1
  12. 2026Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretrainingcited by 1
  13. 2026Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Modelscited by 1
  14. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  15. 2026Test, then Route: How Language Models Execute In-Context Conditional Rules Across Models and Languagescited by 1
  16. 2026SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metricscited by 0
  17. 2026Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Headscited by 0
  18. 2026CoSToM:Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language Modelscited by 0
  19. 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
  20. 2026Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translationcited by 0
  21. 2026DEPART: DEcomposing PARiTy across Multilingual LLMscited by 0
  22. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptationcited by 0
  23. 2026Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Formcited by 0
  24. 2026Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languagescited by 0
  25. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?cited by 0
  26. 2026The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Spacecited by 0
  27. 2026Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMscited by 0
  28. 2026Tracing Stereotypes from Representation to Output in Multilingual LLMscited by 0
  29. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0
  30. 2026Multilinguality in Hybrid Attention LLMscited by 0
  31. 2026Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Contextcited by 0
  32. 2026LLMs program using Reusable and Composable Abstract Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
  2. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages
  3. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
  4. 2025Do Multilingual LLMs Think In English?
  5. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
  6. 2025Improving Multilingual Language Models by Aligning Representations through Steering
  7. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steering
  8. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Models