mlsanityData to 14 Oct 2026

Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks

arXiv 2510.15115 ↗First posted 16 Oct 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of the named entities inserted in the sentence. This leads to numerous instances of ungrammaticality or wrong wording of the final prompts, which complicates the interpretation of scores, especially for languages that have a rich morphological inventory. In this work, we sample 4 Slavic languages from the MLAMA dataset and compare the knowledge retrieval scores between the initial (templated) MLAMA dataset and its sentence-level translations made by Google Translate and ChatGPT. We observe a significant increase in knowledge retrieval scores, and provide a qualitative analysis for possible reasons behind it. We also make an additional analysis of 5 more languages from different families and see similar patterns. Therefore, we encourage the community to control the grammaticality of highly multilingual datasets for higher and more interpretable results, which is well approximated by whole sentence translation with neural MT or LLM systems. The dataset and all related code is published at the Github repository: https://github.com/ZurichNLP/Fluent-mLAMA.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

29 of 45 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  7. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  8. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  9. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  10. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  11. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  12. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
17 more
  1. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  2. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  3. 2021A Simple Recipe for Multilingual Grammatical Error Correctioncited by 108
  4. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  5. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  6. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  7. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  8. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 38
  9. 2019Pushing the Limits of Low-Resource Morphological Inflectioncited by 29
  10. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  11. 2022Language Anisotropic Cross-Lingual Model Editingcited by 26
  12. 2024Cross-lingual Editing in Multilingual Language Modelscited by 18
  13. 2023DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Modelscited by 17
  14. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  15. 2024BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learningcited by 13
  16. 2023Cross-Lingual Knowledge Editing in Large Language Modelscited by 11
  17. 20241+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandiccited by 0
  2. 2026Probing Factual Knowledge Transfer with Training Data Interventionscited by 0
  3. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0