mlsanityData to 14 Oct 2026

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

arXiv 2411.04986 ↗First posted 7 Nov 2024cs.CLcited by 69

Not on the map. Why?

Abstract (arXiv)

Modern language models can process inputs across diverse languages and modalities. We hypothesize that models acquire this capability through learning a shared representation space across heterogeneous data types (e.g., different languages and modalities), which places semantically similar inputs near one another, even if they are from different modalities/languages. We term this the semantic hub hypothesis, following the hub-and-spoke model from neuroscience (Patterson et al., 2007) which posits that semantic knowledge in the human brain is organized through a transmodal semantic "hub" which integrates information from various modality-specific "spokes" regions. We first show that model representations for semantically equivalent inputs in different languages are similar in the intermediate layers, and that this space can be interpreted using the model's dominant pretraining language via the logit lens. This tendency extends to other data types, including arithmetic expressions, code, and visual/audio inputs. Interventions in the shared representation space in one data type also predictably affect model outputs in other data types, suggesting that this shared representations space is not simply a vestigial byproduct of large-scale training on broad data, but something that is actively utilized by the model during input processing.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 55 of its 85 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

55 of 85 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
38 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  3. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  7. 2017Word Translation Without Parallel Datacited by 487
  8. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  9. 2024The Platonic Representation Hypothesiscited by 382
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  11. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  12. 2023In-Context Learning Creates Task Vectorscited by 312
  13. 2023Function Vectors in Large Language Modelscited by 308
  14. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  15. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  16. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  17. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  18. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  19. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  20. 2023Listen, Think, and Understandcited by 209
  21. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  22. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  23. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  24. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  25. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  26. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  27. 2024How do Large Language Models Handle Multilingualism?cited by 150
  28. 2020The Multilingual Amazon Reviews Corpuscited by 143
  29. 2022Linearly Mapping from Image to Text Spacecited by 142
  30. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  31. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 126
  32. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  33. 2023Joint Audio and Speech Understandingcited by 100
  34. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  35. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
  36. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  37. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  38. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 30

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 28; 2026: 38.

69 papers.

Sort:

By area:

  1. 2025Do Multilingual LLMs Think In English?cited by 78
  2. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  3. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  4. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  6. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  7. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  8. 2025LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Modelscited by 23
  9. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 21
  10. 2025Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipelinecited by 21
  11. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 21
  12. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
38 more
  1. 2025Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?cited by 20
  2. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasonerscited by 16
  3. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  4. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  5. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steeringcited by 10
  6. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 9
  7. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 9
  8. 2026Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignmentcited by 9
  9. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 9
  10. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 7
  11. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  12. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Modelscited by 7
  13. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  14. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 6
  15. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  16. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  17. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 5
  18. 2025Language models align with brain regions that represent concepts across modalitiescited by 4
  19. 2025Unsupervised decoding of encoded reasoning using language model interpretabilitycited by 4
  20. 2025Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learningcited by 3
  21. 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden Statescited by 3
  22. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  23. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 2
  24. 2025The Generalization Ridge: Information Flow in Natural Language Generationcited by 2
  25. 2025Mapping Overlaps in Benchmarks through Perplexity in the Wildcited by 2
  26. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 2
  27. 2026Low-Resource Safety Failures Are Action Failures, Not Representation Failurescited by 2
  28. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  29. 2025On Representational Dissociation of Language and Arithmetic in Large Language Modelscited by 1
  30. 2025What if I ask in textitalia lingua? Measuring Functional Similarity Across Languagescited by 1
  31. 2025Neuron-Level Analysis of Cultural Understanding in Large Language Modelscited by 1
  32. 2026Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languagescited by 1
  33. 2026Cross-Modal Taxonomic Generalization in (Vision-) Language Modelscited by 1
  34. 2026Inference-Time Steering for Cross-Lingual Factual Consistency in LLMscited by 1
  35. 2026Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflictscited by 1
  36. 2025Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Modelscited by 0
  37. 2026On the Robustness of Knowledge Editing for Detoxificationcited by 0
  38. 2026Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoderscited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-lingual brain-language model alignment is robust but challenges hierarchical and computational accounts
  2. 2026Abstraction Induces the Brain Alignment of Language and Speech Models
  3. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage
  4. 2024What Makes Two Language Models Think Alike?
  5. 2026Cognitive Expert Language Models Better Align with the Corresponding Brain Systems
  6. 2025Neural Correlates of Language Models Are Specific to Human Language
  7. 2026Abstract representational geometry supports inference in large language models
  8. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units