mlsanityData to 14 Oct 2026

FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality

arXiv 2508.00109 ↗First posted 31 Jul 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Long-form factuality evaluation assesses the ability of models to generate accurate, comprehensive responses to short prompts. Existing benchmarks often lack human verification, leading to potential quality issues. To address this limitation, we introduce FACTORY, a large-scale, human-verified prompt set. Developed using a model-in-the-loop approach and refined by humans, FACTORY includes challenging prompts that are fact-seeking, answerable, and unambiguous. We conduct human evaluations on 6 state-of-the-art language models using FACTORY and existing datasets. Our results show that FACTORY is a challenging benchmark: approximately 40% of the claims made in the responses of SOTA models are not factual, compared to only 10% for other datasets. Our analysis identifies the strengths of FACTORY over prior benchmarks, emphasizing its reliability and the necessity for models to reason across long-tailed facts.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 25 of its 29 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

25 of 29 references.

References · 25

  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2024Measuring short-form factuality in large language modelscited by 373
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
13 more
  1. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  2. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  3. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  4. 2024Long-form factuality in large language modelscited by 155
  5. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  6. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  7. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
  8. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  9. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  10. 2025HALoGEN: Fantastic LLM Hallucinations and Where to Find Themcited by 34
  11. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 25
  12. 2024Improving Model Factuality with Fine-grained Critique-based Evaluatorcited by 15
  13. 2024Improving Factuality with Explicit Working Memorycited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Learning to Reason for Factualitycited by 24
  2. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 4
  3. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  4. 2026Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMscited by 0
  5. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0
  6. 2026Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparisoncited by 0