mlsanityData to 7 Oct 2026

How Language-Neutral is Multilingual BERT?

arXiv 1911.03310 ↗First posted 8 Nov 2019cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Multilingual BERT (mBERT) provides sentence representations for 104 languages, which are useful for many multi-lingual tasks. Previous work probed the cross-linguality of mBERT using zero-shot transfer learning on morphological and syntactic tasks. We instead focus on the semantic properties of mBERT. We show that mBERT representations can be split into a language-specific component and a language-neutral component, and that the language-neutral component is sufficiently general in terms of modeling semantics to allow high-accuracy word-alignment and sentence retrieval but is not yet good enough for the more difficult task of MT quality estimation. Our work presents interesting challenges which must be solved to build better language-neutral representations, particularly for tasks requiring linguistic transfer of semantics.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 7; 2023: 12; 2024: 8; 2025: 5; 2026: 2.

51 papers.

Sort:

By area:

  1. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  2. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  3. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  4. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  5. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  6. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  7. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  8. 2022Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Modelscited by 32
  9. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  10. 2021A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representationscited by 16
  11. 2023Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniquescited by 14
  12. 2021Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesiscited by 13
38 more
  1. 2022KenSwQuAD -- A Question Answering Dataset for Swahili Low Resource Languagecited by 12
  2. 2023Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networkscited by 12
  3. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  4. 2022Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfercited by 11
  5. 2023Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmarkcited by 10
  6. 2021Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Miningcited by 9
  7. 2021Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languagescited by 8
  8. 2021On the ability of monolingual models to learn language-agnostic representationscited by 5
  9. 2022The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfercited by 5
  10. 2023A Joint Matrix Factorization Analysis of Multilingual Representationscited by 5
  11. 2024ChitroJera: A Regionally Relevant Visual Question Answering Dataset for Banglacited by 5
  12. 2024A Truly Joint Neural Architecture for Segmentation and Parsingcited by 4
  13. 2025AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Textcited by 4
  14. 2024Signformer is all you need: Towards Edge AI for Sign Languagecited by 3
  15. 2021Hostility Detection and Covid-19 Fake News Detection in Social Mediacited by 2
  16. 2021Deception detection in text and its relation to the cultural dimension of individualism/collectivismcited by 2
  17. 2021PSG: Prompt-based Sequence Generation for Acronym Extractioncited by 2
  18. 2023Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representationscited by 2
  19. 2023The Less the Merrier? Investigating Language Representation in Multilingual Modelscited by 2
  20. 2024Stanceosaurus 2.0: Classifying Stance Towards Russian and Spanish Misinformationcited by 2
  21. 2026Multilingual Language Models Encode Script Over Linguistic Structurecited by 2
  22. 2021Vy=akarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languagescited by 1
  23. 2021Uppsala NLP at SemEval-2021 Task 2: Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguationcited by 1
  24. 2021Transfer Learning for Multi-lingual Tasks -- a Surveycited by 1
  25. 2021Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Predictioncited by 1
  26. 2024Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfercited by 1
  27. 2025Enhancing Multi-Label Emotion Analysis and Corresponding Intensities for Ethiopian Languagescited by 1
  28. 2025SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation Systemcited by 1
  29. 2021Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?cited by 0
  30. 2022Improving Persian Relation Extraction Models by Data Augmentationcited by 0
  31. 2022Third-Party Aligner for Neural Word Alignmentscited by 0
  32. 2023Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classificationcited by 0
  33. 2023Knowledge Graph-Augmented Korean Generative Commonsense Reasoningcited by 0
  34. 2023Colloquial Persian POS (CPPOS) Corpus: A Novel Corpus for Colloquial Persian Part of Speech Taggingcited by 0
  35. 2024Enhancing Context Through Contrastcited by 0
  36. 2024Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsingcited by 0
  37. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translationcited by 0
  38. 2025Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languagescited by 0

50 most cited of 51 All 51 (JSON)