# AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

arXiv: https://arxiv.org/abs/2402.11073 · HTML: https://mlsanity.com/paper/2402.11073/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society](https://mlsanity.com/paper/2005.00033/) (2020)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [What is the Essence of a Claim? Cross-Domain Claim Identification](https://mlsanity.com/paper/1704.07203/) (2017)
- [IMHO Fine-Tuning Improves Claim Detection](https://mlsanity.com/paper/1905.07000/) (2019)
- [A Benchmark Dataset of Check-worthy Factual Claims](https://mlsanity.com/paper/2004.14425/) (2020)
- [Overview of the CLEF--2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News](https://mlsanity.com/paper/2109.12987/) (2021)
- [LESA: Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content](https://mlsanity.com/paper/2101.11891/) (2021)
- [NewsClaims: A New Benchmark for Claim Detection from News with Attribute Knowledge](https://mlsanity.com/paper/2112.08544/) (2021)
- [Claim Detection in Biomedical Twitter Posts](https://mlsanity.com/paper/2104.11639/) (2021)

## Built on by (most cited first, 28 of 28)

- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024)
- [Towards Effective Extraction and Evaluation of Factual Claims](https://mlsanity.com/paper/2502.10855/) (2025)
- [Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?](https://mlsanity.com/paper/2404.12174/) (2024)
- [A Little Human Data Goes A Long Way](https://mlsanity.com/paper/2410.13098/) (2024)
- [Fake News Detection: Comparative Evaluation of BERT-like Models and Large Language Models with Generative AI-Annotated Data](https://mlsanity.com/paper/2412.14276/) (2024)
- [LLMTaxo: Leveraging Large Language Models for Constructing Taxonomy of Factual Claims from Social Media](https://mlsanity.com/paper/2504.12325/) (2025)
- [Demo: Healthcare Agent Orchestrator (HAO) for Patient Summarization in Molecular Tumor Boards](https://mlsanity.com/paper/2509.06602/) (2025)
- [Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models](https://mlsanity.com/paper/2412.03589/) (2024)
- [Sword and Shield: Uses and Strategies of LLMs in Navigating Disinformation](https://mlsanity.com/paper/2506.07211/) (2025)
- [Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality](https://mlsanity.com/paper/2509.23765/) (2025)
- [DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation](https://mlsanity.com/paper/2406.14162/) (2024)
- [Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification](https://mlsanity.com/paper/2507.05010/) (2025)
- [PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claims](https://mlsanity.com/paper/2504.09866/) (2025)
- [MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations](https://mlsanity.com/paper/2602.05692/) (2026)
- [RAVE: Retrieval and Scoring Aware Verifiable Claim Detection](https://mlsanity.com/paper/2509.15793/) (2025)
- [The Psychology of Falsehood: A Human-Centric Survey of Misinformation Detection](https://mlsanity.com/paper/2509.15896/) (2025)
- [The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation](https://mlsanity.com/paper/2606.03032/) (2026)
- [Efficient Environmental Claim Detection with Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/2502.13628/) (2025)
- [JointCQ: Improving Factual Hallucination Detection with Joint Claim and Query Generation](https://mlsanity.com/paper/2510.19310/) (2025)
- [Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety](https://mlsanity.com/paper/2512.06227/) (2025)
- [SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation](https://mlsanity.com/paper/2602.10017/) (2026)
- [MultiCW: A Large-Scale Balanced Benchmark Dataset for Training Robust Check-Worthiness Detection Models](https://mlsanity.com/paper/2602.16298/) (2026)
- [AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions](https://mlsanity.com/paper/2603.01773/) (2026)
- [ContextClaim: A Context-Driven Paradigm for Verifiable Claim Detection](https://mlsanity.com/paper/2603.30025/) (2026)
- [ToE: A Hierarchical and Explainable Claim Verification Framework with Dynamic Multi-source Evidence Retrieval and Aggregation](https://mlsanity.com/paper/2606.27736/) (2026)
- [RCMN: Understanding Misleadingness in Influential Public Discourse](https://mlsanity.com/paper/2608.27358/) (2026)
- [Calibrating Small Language Models for Claim Check-Worthiness Detection](https://mlsanity.com/paper/2608.30731/) (2026)

## Abstract (arXiv)

With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.