mlsanityData to 7 Oct 2026

Long-form factuality in large language models

arXiv 2403.18802 ↗First posted 27 Mar 2024cs.CL cs.AI cs.LGcited by 146

Not on the map. Why?

Cited by 23 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall). Empirically, we demonstrate that LLM agents can outperform crowdsourced human annotators - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Mar 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 52; 2026: 54.

146 papers.

Sort:

By area:

  1. 2024Measuring short-form factuality in large language modelscited by 345
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  4. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  5. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  6. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  7. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  8. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  9. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  10. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  11. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  12. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
38 more
  1. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 44
  2. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
  3. 2024FIRE: Fact-checking with Iterative Retrieval and Verificationcited by 39
  4. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  5. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  6. 2024GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Frameworkcited by 30
  7. 2024OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMscited by 28
  8. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  9. 2024RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Modelscited by 24
  10. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 23
  11. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 23
  12. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 22
  13. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  14. 2024Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detectorcited by 20
  15. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 20
  16. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  17. 2025OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinkingcited by 18
  18. 2024Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?cited by 17
  19. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMscited by 17
  20. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  21. 2024Loki: An Open-Source Tool for Fact Verificationcited by 16
  22. 2024Core: Robust Factual Precision with Informative Sub-Claim Identificationcited by 15
  23. 2025Towards Personalized Deep Research: Benchmarks and Evaluationscited by 14
  24. 2024OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMscited by 13
  25. 2024Multi-group Uncertainty Quantification for Long-form Text Generationcited by 13
  26. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 13
  27. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  28. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  29. 2024DebateQA: Evaluating Question Answering on Debatable Knowledgecited by 12
  30. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 12
  31. 2025MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimizationcited by 12
  32. 2025JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoringcited by 12
  33. 2025AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Modelscited by 12
  34. 2024TapeAgents: a Holistic Framework for Agent Development and Optimizationcited by 11
  35. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  36. 2024The Factuality of Large Language Models in the Legal Domaincited by 10
  37. 2024A Guide to Misinformation Detection Data and Evaluationcited by 10
  38. 2024Improving Factuality with Explicit Working Memorycited by 10

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Fine-tuning Language Models for Factuality
  2. 2024UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models
  3. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
  4. 2024Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output
  5. 2024FactAlign: Long-form Factuality Alignment of Large Language Models
  6. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
  7. 2025Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation
  8. 2024Identifying the Achilles' Heel: An Iterative Method for Dynamically Uncovering Factual Errors in Large Language Models