# The Case for Claim Difficulty Assessment in Automatic Fact Checking

arXiv: https://arxiv.org/abs/2109.09689 · HTML: https://mlsanity.com/paper/2109.09689/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning](https://mlsanity.com/paper/1809.06416/) (2018)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [X-FACT: A New Benchmark Dataset for Multilingual Fact Checking](https://mlsanity.com/paper/2106.09248/) (2021)
- [A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking](https://mlsanity.com/paper/1911.01214/) (2019)
- [Integrating Stance Detection and Fact Checking in a Unified Corpus](https://mlsanity.com/paper/1804.08012/) (2018)
- [Confidence Modeling for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04604/) (2018)
- [A Review of Winograd Schema Challenge Datasets and Approaches](https://mlsanity.com/paper/2004.13831/) (2020)
- [CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media](https://mlsanity.com/paper/2001.08546/) (2020)
- [Joint Verification and Reranking for Open Fact Checking Over Tables](https://mlsanity.com/paper/2012.15115/) (2020)
- [Long-length Legal Document Classification](https://mlsanity.com/paper/1912.06905/) (2019)

## Built on by (most cited first, 5 of 5)

- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [Contrastive Learning to Improve Retrieval for Real-world Fact Checking](https://mlsanity.com/paper/2410.04657/) (2024)

## Abstract (arXiv)

Fact-checking is the process of evaluating the veracity of claims (i.e., purported facts). In this opinion piece, we raise an issue that has received little attention in prior work -- that some claims are far more difficult to fact-check than others. We discuss the implications this has for both practical fact-checking and research on automated fact-checking, including task formulation and dataset design. We report a manual analysis undertaken to explore factors underlying varying claim difficulty and identify several distinct types of difficulty. We motivate this new claim difficulty prediction task as beneficial to both automated fact-checking and practical fact-checking organizations.