mlsanityData to 7 Oct 2026

Factual Consistency of Multilingual Pretrained Language Models

arXiv 2203.11552 ↗First posted 22 Mar 2022cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

24 of 39 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
12 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  3. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  4. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  5. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  6. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  7. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  8. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  9. 2019Inducing Relational Knowledge from BERTcited by 75
  10. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  11. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  12. 2021Accurate, yet inconsistent? Consistency Analysis on Language Understanding Modelscited by 13

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 8; 2024: 2; 2025: 4.

15 papers.

Sort:

By area:

  1. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  2. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  3. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  4. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  5. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  6. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  7. 2023DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Modelscited by 17
  8. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  9. 2023Semantic Consistency for Assuring Reliability of Large Language Modelscited by 15
  10. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  11. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  12. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
3 more
  1. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  2. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  3. 2025FIBER: A Multilingual Evaluation Resource for Factual Inference Biascited by 0