# OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs

arXiv: https://arxiv.org/abs/2405.05583 · HTML: https://mlsanity.com/paper/2405.05583/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 17 of 17)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Generating Benchmarks for Factuality Evaluation of Language Models](https://mlsanity.com/paper/2307.06908/) (2023)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [Neural Text Generation from Structured Data with Application to the Biography Domain](https://mlsanity.com/paper/1603.07771/) (2016)

## Built on by (most cited first, 13 of 13)

- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking](https://mlsanity.com/paper/2510.13842/) (2025)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [If We May De-Presuppose: Robustly Verifying Claims through Presupposition-Free Question Decomposition](https://mlsanity.com/paper/2508.16838/) (2025)
- [UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking](https://mlsanity.com/paper/2505.15063/) (2025)
- [Task-Oriented Automatic Fact-Checking with Frame-Semantics](https://mlsanity.com/paper/2501.13288/) (2025)
- [Fine-grained Claim-level RAG Benchmark for Law](https://mlsanity.com/paper/2605.21071/) (2026)
- [CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval](https://mlsanity.com/paper/2601.17230/) (2026)
- [Fact-Checking with Large Language Models via Probabilistic Certainty and Consistency](https://mlsanity.com/paper/2601.02574/) (2026)
- [TAIGR: Towards Modeling Influencer Content on Social Media via Structured, Pragmatic Inference](https://mlsanity.com/paper/2601.20032/) (2026)
- [ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models](https://mlsanity.com/paper/2603.15513/) (2026)
- [Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs](https://mlsanity.com/paper/2606.21359/) (2026)
- [MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification](https://mlsanity.com/paper/2609.12884/) (2026)

## Abstract (arXiv)

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. Difficulties lie in assessing the factuality of free-form responses in open domains. Also, different papers use disparate evaluation benchmarks and measurements, which renders them hard to compare and hampers future progress. To mitigate these issues, we propose OpenFactCheck, a unified framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. OpenFactCheck consists of three modules: (i) CUSTCHECKER allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims, (ii) LLMEVAL, a unified evaluation framework assesses LLM's factuality ability from various perspectives fairly, and (iii) CHECKEREVAL is an extensible solution for gauging the reliability of automatic fact-checkers' verification results using human-annotated datasets. Data and code are publicly available at https://github.com/yuxiaw/openfactcheck.