# Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification

arXiv: https://arxiv.org/abs/2402.10735 · HTML: https://mlsanity.com/paper/2402.10735/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 26 of its 92 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples](https://mlsanity.com/paper/2305.15269/) (2023)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2402.08955/) (2024)
- [FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking](https://mlsanity.com/paper/2309.00240/) (2023)
- [Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models](https://mlsanity.com/paper/2305.14623/) (2023)
- [Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability](https://mlsanity.com/paper/2401.08574/) (2024)
- [GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach](https://mlsanity.com/paper/2308.09267/) (2023)
- [The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models](https://mlsanity.com/paper/2401.12117/) (2024)
- [Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition](https://mlsanity.com/paper/2311.09552/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision](https://mlsanity.com/paper/2505.20415/) (2025)
- [Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs](https://mlsanity.com/paper/2604.08016/) (2026)
- [ExVerus: Verus Proof Repair via Counterexample Reasoning](https://mlsanity.com/paper/2603.25810/) (2026)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning](https://mlsanity.com/paper/2510.17590/) (2025)
- [MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning](https://mlsanity.com/paper/2601.10649/) (2026)
- [The Alignment Bottleneck in Decomposition-Based Claim Verification](https://mlsanity.com/paper/2602.10380/) (2026)
- [Quantum Knowledge Graph: Modeling Context-Dependent Triplet Validity](https://mlsanity.com/paper/2604.23972/) (2026)

## Abstract (arXiv)

Although LLMs have shown great performance on Mathematics and Coding related reasoning tasks, the reasoning capabilities of LLMs regarding other forms of reasoning are still an open problem. Here, we examine the issue of reasoning from the perspective of claim verification. We propose a framework designed to break down any claim paired with evidence into atomic reasoning types that are necessary for verification. We use this framework to create RECV, the first claim verification benchmark, incorporating real-world claims, to assess the deductive and abductive reasoning capabilities of LLMs. The benchmark comprises of three datasets, covering reasoning problems of increasing complexity. We evaluate three state-of-the-art proprietary LLMs under multiple prompt settings. Our results show that while LLMs can address deductive reasoning problems, they consistently fail in cases of abductive reasoning. Moreover, we observe that enhancing LLMs with rationale generation is not always beneficial. Nonetheless, we find that generated rationales are semantically similar to those provided by humans, especially in deductive reasoning cases.