mlsanityData to 14 Oct 2026

It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT

arXiv 2010.08275 ↗First posted 16 Oct 2020cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Recent works have demonstrated that multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages. We study the word-level translation information embedded in mBERT and present two simple methods that expose remarkable translation capabilities with no fine-tuning. The results suggest that most of this information is encoded in a non-linear way, while some of it can also be recovered with purely linear tools. As part of our analysis, we test the hypothesis that mBERT learns representations which contain both a language-encoding component and an abstract, cross-lingual component, and explicitly identify an empirical language-identity subspace within mBERT representations.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 7; 2024: 4; 2025: 4; 2026: 1.

30 papers.

Sort:

  1. 2022Linear Adversarial Concept Erasurecited by 101
  2. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  3. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  4. 2022Kernelized Concept Erasurecited by 48
  5. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  6. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  7. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  8. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 19
  9. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  10. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  11. 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
  12. 2022Analyzing Gender Representation in Multilingual Modelscited by 10
18 more
  1. 2022Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?cited by 10
  2. 2021On the Language-specificity of Multilingual BERT and the Impact of Fine-tuningcited by 8
  3. 2022Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoderscited by 7
  4. 2022Improving Bilingual Lexicon Induction with Cross-Encoder Rerankingcited by 7
  5. 2021Locating Language-Specific Information in Contextualized Embeddingscited by 6
  6. 2025Fundamental Limits of Perfect Concept Erasurecited by 6
  7. 2023When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon Induction for Related Data-Imbalanced Languagescited by 3
  8. 2022MATra: A Multilingual Attentive Transliteration System for Indian Scriptscited by 2
  9. 2023Cross-Lingual Transfer of Cognitive Processing Complexitycited by 2
  10. 2023Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Modelscited by 2
  11. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  12. 2023Counterfactually Probing Language Identity in Multilingual Modelscited by 2
  13. 2025InstructNet: A Novel Approach for Multi-Label Instruction Classification through Advanced Deep Learningcited by 2
  14. 2024MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detectioncited by 1
  15. 2024A Systematic Survey of Natural Language Processing for the Greek Languagecited by 1
  16. 2025Static Word Embeddings for Sentence Semantic Representationcited by 1
  17. 2025MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Languagecited by 1
  18. 2026Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Modelscited by 0