# Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection

arXiv: https://arxiv.org/abs/1809.08193 · HTML: https://mlsanity.com/paper/1809.08193/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2018.

## Built on by (most cited first, 30 of 55)

- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains](https://mlsanity.com/paper/2402.00559/) (2024)
- [Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2109.11427/) (2021)
- [Claim Matching Beyond English to Scale Global Fact-Checking](https://mlsanity.com/paper/2106.00853/) (2021)
- [Claim Detection for Automated Fact-checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research](https://mlsanity.com/paper/2401.11969/) (2024)
- [AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators](https://mlsanity.com/paper/2402.11073/) (2024)
- [Logical Fallacy Detection](https://mlsanity.com/paper/2202.13758/) (2022)
- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document](https://mlsanity.com/paper/2109.07410/) (2021)
- [The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who](https://mlsanity.com/paper/2304.14238/) (2023)
- [Towards Effective Extraction and Evaluation of Factual Claims](https://mlsanity.com/paper/2502.10855/) (2025)
- [From Chaos to Clarity: Claim Normalization to Empower Fact-Checking](https://mlsanity.com/paper/2310.14338/) (2023)
- [LAraBench: Benchmarking Arabic AI with Large Language Models](https://mlsanity.com/paper/2305.14982/) (2023)
- [Matching Tweets With Applicable Fact-Checks Across Languages](https://mlsanity.com/paper/2202.07094/) (2022)
- [Findings of the NLP4IF-2021 Shared Tasks on Fighting the COVID-19 Infodemic and Censorship Detection](https://mlsanity.com/paper/2109.12986/) (2021)
- [Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments](https://mlsanity.com/paper/2212.09683/) (2022)
- [SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval](https://mlsanity.com/paper/2505.10740/) (2025)
- [Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms](https://mlsanity.com/paper/2204.13568/) (2022)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [Claim Check-Worthiness Detection: How Well do LLMs Grasp Annotation Guidelines?](https://mlsanity.com/paper/2404.12174/) (2024)
- [Native vs Non-Native Language Prompting: A Comparative Analysis](https://mlsanity.com/paper/2409.07054/) (2024)
- [FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learning](https://mlsanity.com/paper/2109.03094/) (2021)
- [Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systems](https://mlsanity.com/paper/2211.12143/) (2022)
- [Can LLMs Automate Fact-Checking Article Writing?](https://mlsanity.com/paper/2503.17684/) (2025)
- [Document-level Claim Extraction and Decontextualisation for Fact-Checking](https://mlsanity.com/paper/2406.03239/) (2024)
- [Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents](https://mlsanity.com/paper/2507.19090/) (2025)
- [HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims](https://mlsanity.com/paper/2502.11753/) (2025)
- [Attacks by Content: Automated Fact-checking is an AI Security Issue](https://mlsanity.com/paper/2510.11238/) (2025)
- [Self-Supervised Claim Identification for Automated Fact Checking](https://mlsanity.com/paper/2102.02335/) (2021)

## Abstract (arXiv)

In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.