mlsanityData to 7 Oct 2026

The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

arXiv 2604.00019 ↗First posted 11 Mar 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

21 of 29 references.

References · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  12. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
9 more
  1. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  2. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  3. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  4. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
  5. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  6. 2024Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScorecited by 18
  7. 2025Towards Effective Extraction and Evaluation of Factual Claimscited by 18
  8. 2024Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Factscited by 14
  9. 2025The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answerscited by 2

Built on by

Not cited yet.