mlsanityData to 7 Oct 2026

An Isotropy Analysis in the Multilingual BERT Embedding Space

arXiv 2110.04504 ↗First posted 9 Oct 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outlier dimensions. We show that, unlike its monolingual counterpart, the multilingual BERT model exhibits no outlier dimension in its representations while it has a highly anisotropic space. There are a few dimensions in the monolingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our analysis indicates that, despite having different degenerated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

20 of 29 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  7. 2019How multilingual is Multilingual BERT?cited by 642
  8. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  9. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  10. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  11. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  12. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
8 more
  1. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  2. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  3. 2018Decorrelated Batch Normalizationcited by 83
  4. 2020Are All Good Word Vector Spaces Isomorphic?cited by 36
  5. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  6. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 30
  7. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  8. 2021Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 7; 2025: 3; 2026: 6.

25 papers.

Sort:

  1. 2022The Geometry of Multilingual Language Model Representationscited by 94
  2. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  3. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  4. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  5. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  6. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  7. 2023Byte Pair Encoding for Symbolic Musiccited by 19
  8. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  9. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  10. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  11. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 9
  12. 2022IsoVec: Controlling the Relative Isomorphism of Word Embedding Spacescited by 7
13 more
  1. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  2. 2022A Natural Bias for Language Generation Modelscited by 6
  3. 2023Is Anisotropy Inherent to Transformers?cited by 5
  4. 2023The Less the Merrier? Investigating Language Representation in Multilingual Modelscited by 2
  5. 2025CodeSCM: Causal Analysis for Multi-Modal Code Generationcited by 2
  6. 2025Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfercited by 2
  7. 2026Weight Tying Biases Token Embeddings Towards the Output Spacecited by 2
  8. 2025CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurementcited by 1
  9. 2026Pingala: Prosody-Aware Decoding for Sanskrit Poetry Generationcited by 1
  10. 2026Capacity Constraints and the Multilingual Penalty for Lexical Disambiguationcited by 0
  11. 2026Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometrycited by 0
  12. 2026Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Testscited by 0
  13. 2026The Geometry of Low-Resource Language Representationscited by 0