mlsanityData to 7 Oct 2026

The Geometry of Multilingual Language Model Representations

arXiv 2205.10964 ↗First posted 22 May 2022cs.CLcited by 94

Not on the map. Why?

Abstract (arXiv)

We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages. The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies. Shifting representations by language means is sufficient to induce token predictions in different languages. However, we also identify stable language-neutral axes that encode information such as token positions and part-of-speech. We visualize representations projected onto language-sensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information. These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

22 of 41 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2019How multilingual is Multilingual BERT?cited by 642
  8. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  10. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  11. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  12. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
10 more
  1. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  2. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  3. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  4. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  5. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  6. 2020Probing Multilingual BERT for Genetic and Typological Signalscited by 26
  7. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
  8. 2020Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Studycited by 24
  9. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  10. 2021Locating Language-Specific Information in Contextualized Embeddingscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 24; 2025: 22; 2026: 30.

94 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  5. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  6. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  7. 2023Latent Space Translation via Semantic Alignmentcited by 46
  8. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  9. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  10. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  11. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  12. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
38 more
  1. 2024mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?cited by 20
  2. 2024Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMscited by 20
  3. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  4. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  5. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  6. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  7. 2024The Hidden Space of Transformer Language Adapterscited by 16
  8. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  9. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 15
  10. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  11. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  12. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  13. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  14. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  15. 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
  16. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steeringcited by 12
  17. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learnerscited by 11
  18. 2024Latent Space Translation via Inverse Relative Projectioncited by 10
  19. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  20. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 9
  21. 2023Representation Of Lexical Stylistic Features In Language Models' Embedding Spacecited by 8
  22. 2023Second Language Acquisition of Neural Language Modelscited by 8
  23. 2023GradSim: Gradient-Based Language Grouping for Effective Multilingual Trainingcited by 6
  24. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 6
  25. 2022Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncodercited by 5
  26. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  27. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  28. 2025What Do Large Language Models Know? Tacit Knowledge as a Potential Causal-Explanatory Structurecited by 5
  29. 2025Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representationscited by 5
  30. 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agentscited by 5
  31. 2026Curveball Steering: The Right Direction To Steer Isn't Always Linearcited by 5
  32. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4
  33. 2025Connecting Neural Models Latent Geometries with Relative Geodesic Representationscited by 4
  34. 2025Language Surgery in Multilingual Large Language Modelscited by 4
  35. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  36. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  37. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  38. 2025Analyzing the relationships between pretraining language, phonetic, tonal, and speaker information in self-supervised speech modelscited by 3

50 most cited of 94 All 94 (JSON)