mlsanityData to 13 Oct 2026

Locating Language-Specific Information in Contextualized Embeddings

arXiv 2109.08040 ↗First posted 16 Sep 2021cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages. Most MPLMs are trained in an unsupervised fashion and the relationship between their objective and multilinguality is unclear. More specifically, the question whether MPLM representations are language-agnostic or they simply interleave well with learned task prediction heads arises. In this work, we locate language-specific information in MPLMs and identify its dimensionality and the layers where this information occurs. We show that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. Our study contributes to a better understanding of MPLM representations, going beyond treating them as unanalyzable blobs of information.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

14 of 22 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 741
  4. 2019How multilingual is Multilingual BERT?cited by 643
  5. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  6. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  7. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 83
  8. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  9. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  10. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  11. 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
  12. 2021Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Modelscited by 22
2 more
  1. 2019Analytical Methods for Interpretable Ultradense Word Embeddingscited by 13
  2. 2016Ultradense Word Embeddings by Orthogonal Transformationcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 1.

6 papers.

  1. 2022The Geometry of Multilingual Language Model Representationscited by 94
  2. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  3. 2023Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understandingcited by 7
  4. 2023A Joint Matrix Factorization Analysis of Multilingual Representationscited by 5
  5. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  6. 2023mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Modelscited by 0