# OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

arXiv: https://arxiv.org/abs/2408.11832 · HTML: https://mlsanity.com/paper/2408.11832/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Generating Benchmarks for Factuality Evaluation of Language Models](https://mlsanity.com/paper/2307.06908/) (2023)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Neural Text Generation from Structured Data with Application to the Biography Domain](https://mlsanity.com/paper/1603.07771/) (2016)

## Built on by (most cited first, 28 of 28)

- [DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts](https://mlsanity.com/paper/2412.10510/) (2024)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?](https://mlsanity.com/paper/2411.02400/) (2024)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Loki: An Open-Source Tool for Fact Verification](https://mlsanity.com/paper/2410.01794/) (2024)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [Optimizing Decomposition for Optimal Claim Verification](https://mlsanity.com/paper/2503.15354/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs](https://mlsanity.com/paper/2510.12839/) (2025)
- [Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models](https://mlsanity.com/paper/2412.15265/) (2024)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models](https://mlsanity.com/paper/2408.15729/) (2024)
- [UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking](https://mlsanity.com/paper/2505.15063/) (2025)
- [Shakespearean Sparks: The Dance of Hallucination and Creativity in LLMs' Decoding Layers](https://mlsanity.com/paper/2503.02851/) (2025)
- [PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claims](https://mlsanity.com/paper/2504.09866/) (2025)
- [MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations](https://mlsanity.com/paper/2505.14101/) (2025)
- [Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine](https://mlsanity.com/paper/2506.20876/) (2025)
- [Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction](https://mlsanity.com/paper/2509.03540/) (2025)
- [Explicit and Implicit Data Augmentation for Social Event Detection](https://mlsanity.com/paper/2509.04202/) (2025)
- [ShortCheck: Checkworthiness Detection of Multilingual Short-Form Videos](https://mlsanity.com/paper/2509.20467/) (2025)
- [AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models](https://mlsanity.com/paper/2506.11110/) (2025)
- [FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts](https://mlsanity.com/paper/2508.00889/) (2025)
- [PledgeTracker: A System for Monitoring the Fulfilment of Pledges](https://mlsanity.com/paper/2509.11804/) (2025)
- [ClaimCheck: Real-Time Fact-Checking with Small Language Models](https://mlsanity.com/paper/2510.01226/) (2025)
- [Human-Anchored Factuality Evaluation with Strategic Annotation](https://mlsanity.com/paper/2609.00494/) (2026)
- [MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification](https://mlsanity.com/paper/2609.12884/) (2026)
- [SoK: Formal Methods for Fact-Checking and Information Integrity](https://mlsanity.com/paper/2609.23239/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)

## Abstract (arXiv)

The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures, which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (https://github.com/mbzuai-nlp/openfactcheck) and publicly released as a Python library (https://pypi.org/project/openfactcheck/) and also as a web service (http://app.openfactcheck.com). A video describing the system is available at https://youtu.be/-i9VKL0HleI.