An Analysis of Multilingual FActScore
Not on the map. Why?
Abstract (arXiv)
FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This paper studies the limitations of each component in the four-component pipeline of FActScore in the multilingual setting. We introduce a new dataset for FActScore on texts generated by strong multilingual LLMs. Our evaluation shows that LLMs exhibit distinct behaviors in both fact extraction and fact scoring tasks. No LLM produces consistent and reliable FActScore across languages with varying levels of resources. We also find that the knowledge source plays an important role in the quality of the estimated FActScore. Using Wikipedia as the knowledge source may hinder the true FActScore of long-form text due to its limited coverage in medium- and low-resource languages. We also incorporate three mitigations to our knowledge source that ultimately improve FActScore estimation across all languages.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 20 Jun 2024.
Builds on
19 of 28 references.
References · 19
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2019Language Models as Knowledge Bases?cited by 1,936
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2024Yi: Open Foundation Models by 01.AIcited by 719
- 2023In-Context Retrieval-Augmented Language Modelscited by 702
- 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
- 2023Effective Long-Context Scaling of Foundation Modelscited by 303
- 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
7 more
- 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
- 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
- 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
- 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
- 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
- 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
- 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 43
Built on by
Citing papers per year: 2025: 2.
2 papers.