mlsanityData to 7 Oct 2026

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

arXiv 2311.09000 ↗First posted 15 Nov 2023cs.CLcited by 86

Not on the map. Why?

Abstract (arXiv)

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

35 of 57 references.

References · 35

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2021Ethical and social risks of harm from Language Modelscited by 957
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  6. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  7. 2021A Survey on Automated Fact-Checkingcited by 376
  8. 2023How Language Model Hallucinations Can Snowballcited by 310
  9. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  10. 2022Teaching language models to support answers with verified quotescited by 290
  11. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  12. 2023Do Large Language Models Know What They Don't Know?cited by 238
23 more
  1. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  2. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  3. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  4. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  5. 2023A Categorical Archive of ChatGPT Failurescited by 167
  6. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  7. 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
  8. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  9. 2020Generating Fact Checking Explanationscited by 132
  10. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  11. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  12. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  13. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  14. 2021Decontextualization: Making Sentences Stand-Alonecited by 101
  15. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  16. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  17. 2020Generating Fact Checking Briefscited by 50
  18. 2020Evidence-based Factual Error Correctioncited by 44
  19. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  20. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  21. 2021Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric Viewcited by 24
  22. 2021Community-Based Fact-Checking on Twitter's Birdwatch Platformcited by 13
  23. 2021Towards Explainable Fact Checkingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 35; 2026: 28.

86 papers.

Sort:

By area:

  1. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  3. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  4. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  5. 2024FIRE: Fact-checking with Iterative Retrieval and Verificationcited by 39
  6. 2024LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentationcited by 28
  7. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  8. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 23
  9. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 23
  10. 2024DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generationcited by 22
  11. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  12. 2024Generative Large Language Models in Automated Fact-Checking: A Surveycited by 19
38 more
  1. 2024A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterizationcited by 18
  2. 2025Towards Effective Extraction and Evaluation of Factual Claimscited by 18
  3. 2024Loki: An Open-Source Tool for Fact Verificationcited by 16
  4. 2024OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMscited by 13
  5. 2024FactLens: Benchmarking Fine-Grained Fact Verificationcited by 10
  6. 2025Optimizing Decomposition for Optimal Claim Verificationcited by 10
  7. 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factualitycited by 10
  8. 2024A Reality Check on Context Utilisation for Retrieval-Augmented Generationcited by 9
  9. 2025RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checkingcited by 9
  10. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?cited by 9
  11. 2024ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertaintycited by 8
  12. 2025BiDeV: Bilateral Defusing Verification for Complex Claim Fact-Checkingcited by 8
  13. 2025VeriFastScore: Speeding up long-form factuality evaluationcited by 8
  14. 2025A Neurosymbolic Approach to Natural Language Formalization and Verificationcited by 8
  15. 2024Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checkingcited by 7
  16. 2025FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMscited by 7
  17. 2025Evaluating open-source Large Language Models for automated fact-checkingcited by 6
  18. 2025The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Paperscited by 6
  19. 2025MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responsescited by 6
  20. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 5
  21. 2025Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generationcited by 5
  22. 2025UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarkingcited by 4
  23. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  24. 2026FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verificationcited by 4
  25. 2024Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Educationcited by 3
  26. 2024Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024cited by 3
  27. 2025SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searchescited by 3
  28. 2025EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actionscited by 3
  29. 2025Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Expertscited by 3
  30. 2025Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledgecited by 3
  31. 2025Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checkingcited by 2
  32. 2025Fact-checking AI-generated news reports: Can LLMs catch their own lies?cited by 2
  33. 2025PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claimscited by 2
  34. 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generationcited by 2
  35. 2025Exposing Citation Vulnerabilities in Generative Enginescited by 2
  36. 2025Stress Testing Factual Consistency Metrics for Long-Document Summarizationcited by 2
  37. 2025Heaven-Sent or Hell-Bent? Benchmarking the Intelligence and Defectiveness of LLM Hallucinationscited by 2
  38. 2025Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Abilitycited by 1

50 most cited of 86 All 86 (JSON)