mlsanityData to 7 Oct 2026

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

arXiv 2102.00894 ↗First posted 1 Feb 2021cs.CLcited by 132

Not on the map. Why?

Abstract (arXiv)

Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 1 Feb 2021.

Builds on

25 of 34 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
13 more
  1. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  2. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  3. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  4. 2020How Context Affects Language Models' Factual Predictionscited by 181
  5. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  6. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  7. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  8. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  9. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  10. 2020Pre-training via Paraphrasingcited by 92
  11. 2019Inducing Relational Knowledge from BERTcited by 75
  12. 2020BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QAcited by 36
  13. 2016Intrinsic Subspace Evaluation of Word Embedding Representationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 18; 2023: 23; 2024: 39; 2025: 25; 2026: 10.

132 papers.

Sort:

By area:

  1. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  2. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  3. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  5. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  6. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  7. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  8. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  9. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  10. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  11. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  12. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
38 more
  1. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  2. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 50
  3. 2023Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Textcited by 48
  4. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  5. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  6. 2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
  7. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  8. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  9. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  10. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  11. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  12. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  13. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  14. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 28
  15. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  16. 2022Language Anisotropic Cross-Lingual Model Editingcited by 26
  17. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  18. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  19. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  20. 2021Dataset Geography: Mapping Language Data to Language Userscited by 22
  21. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  22. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  23. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  24. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  25. 2023A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontierscited by 21
  26. 2024Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMscited by 20
  27. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  28. 2021mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Modelscited by 19
  29. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  30. 2023Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?cited by 19
  31. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  32. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18
  33. 2022Neural Approaches to Multilingual Information Retrievalcited by 18
  34. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  35. 2024Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Modelscited by 18
  36. 2022What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classificationcited by 17
  37. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  38. 2023DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Modelscited by 17

50 most cited of 132 All 132 (JSON)