mlsanityData to 7 Oct 2026

What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis

arXiv 2604.01657 ↗First posted 2 Apr 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 24K claim-verification examples across 9 datasets using GPT-4o-mini and find that direct evidence extraction dominates, while multi-sentence synthesis and numerical reasoning are severely under-represented. A dataset-level breakdown reveals stark biases: some datasets almost exclusively test lexical matching, while others require information synthesis in roughly half of cases. Using a compact 1B-parameter reasoning verifier, we further characterize five error types and show that error profiles vary dramatically by domain -- general-domain verification is dominated by lexical overlap bias, scientific verification by overcautiousness, and mathematical verification by arithmetic reasoning failures. Our findings suggest that high benchmark scores primarily reflect retrieval-plus-entailment ability. We outline recommendations for building more challenging evaluation suites that better test the reasoning capabilities verification systems need.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 21 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

17 of 21 references.

References · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  9. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  10. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  11. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  12. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
5 more
  1. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  2. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  3. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
  4. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  5. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification
  2. 2025Think Right, Not More: Test-Time Scaling for Numerical Claim Verification
  3. 2024Robust Claim Verification Through Fact Detection
  4. 2025Fact in Fragments: Deconstructing Complex Claims via LLM-based Atomic Fact Extraction and Verification
  5. 2025Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs
  6. 2026Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
  7. 2024CHECKWHY: Causal Fact Verification via Argument Structure
  8. 2025STRIVE: Structured Reasoning for Self-Improvement in Claim Verification