mlsanityData to 7 Oct 2026

FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

arXiv 2305.14251 ↗First posted 23 May 2023cs.CL cs.AI cs.LGcited by 1,259

Not on the map. Why?

Cited by 157 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly. In this paper, we introduce FACTSCORE, a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source. We conduct an extensive human evaluation to obtain FACTSCOREs of people biographies generated by several state-of-the-art commercial LMs -- InstructGPT, ChatGPT, and the retrieval-augmented PerplexityAI -- and report new analysis demonstrating the need for such a fine-grained score (e.g., ChatGPT only achieves 58%). Since human evaluation is costly, we also introduce an automated model that estimates FACTSCORE using retrieval and a strong language model, with less than a 2% error rate. Finally, we use this automated metric to evaluate 6,500 generations from a new set of 13 recent LMs that would have cost 26K if evaluated by humans, with various findings: GPT-4 and ChatGPT are more factual than public models, and Vicuna and Alpaca are some of the best public models. FACTSCORE is available for public use via `pip install factscore`.

Key facts. Cited by 1,259 papers in the map (data through 7 Oct 2026). Builds on 52 of its 78 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

52 of 78 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
38 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  2. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  3. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  4. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  6. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  9. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  13. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  14. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  15. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  16. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  17. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  18. 2022Is GPT-3 a Good Data Annotator?cited by 208
  19. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  20. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  21. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  22. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  23. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  24. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  25. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  26. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  27. 2017Generating Sentences by Editing Prototypescited by 88
  28. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  29. 2019Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claimscited by 73
  30. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  31. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  32. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  33. 2020Generating Fact Checking Briefscited by 50
  34. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  35. 2018Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthinesscited by 41
  36. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  37. 2022Nonparametric Masked Language Modelingcited by 39
  38. 2021Paraphrastic Representations at Scalecited by 19

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 75; 2024: 336; 2025: 352; 2026: 496.

1,259 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  7. 2024Large Language Models: A Surveycited by 506
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  12. 2024Measuring short-form factuality in large language modelscited by 345
38 more
  1. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  2. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  5. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  10. 2023Fine-tuning Language Models for Factualitycited by 219
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  12. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  13. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  14. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  15. 2024Corrective Retrieval Augmented Generationcited by 187
  16. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  17. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  18. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  19. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  20. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  21. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  22. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  23. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  24. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  25. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  26. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  27. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  28. 2024Language Models with Conformal Factuality Guaranteescited by 128
  29. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  30. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  31. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  32. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  33. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  34. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  35. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  36. 2024Large Language Models Meet NLP: A Surveycited by 102
  37. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  38. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98

50 most cited of 1,259 All 1,259 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
  2. 2024Long-form factuality in large language models
  3. 2024Face the Facts! Evaluating RAG-based Pipelines for Professional Fact-Checking
  4. 2025FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models
  5. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
  6. 2024Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output
  7. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
  8. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation