mlsanityData to 7 Oct 2026

Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models

arXiv 2305.13675 ↗First posted 23 May 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with counterfactuals, 2) evaluate 5 models in a multilingual test, and 3) benchmark a diverse set of 24 models in an English-only test. Meta's LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA's errors reveals significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. Overall, our findings suggest that today's foundation models are far from polyglots.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

22 of 28 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2022Mass-Editing Memory in a Transformercited by 976
10 more
  1. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  2. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  7. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  8. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  9. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  10. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  3. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  4. 2025Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?cited by 5