# Investigating the Robustness of Deductive Reasoning with Large Language Models

arXiv: https://arxiv.org/abs/2502.04352 · HTML: https://mlsanity.com/paper/2502.04352/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 3 of its 36 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 3 of 3)

- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge](https://mlsanity.com/paper/2311.08147/) (2023)
- [RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models](https://mlsanity.com/paper/2406.11020/) (2024)

## Built on by (most cited first, 8 of 8)

- [ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning](https://mlsanity.com/paper/2512.07795/) (2025)
- [Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models](https://mlsanity.com/paper/2505.23715/) (2025)
- [Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretations](https://mlsanity.com/paper/2507.09751/) (2025)
- [FSLI: An Interpretable Formal Semantic System for One-Dimensional Ordering Inference](https://mlsanity.com/paper/2502.08415/) (2025)
- [A Mathematical Theory of Discursive Networks](https://mlsanity.com/paper/2507.06565/) (2025)
- [Transformers Can Learn Connectivity in Some Graphs but Not Others](https://mlsanity.com/paper/2509.22343/) (2025)
- [Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities](https://mlsanity.com/paper/2608.12374/) (2026)
- [Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning](https://mlsanity.com/paper/2609.28442/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been shown to achieve impressive results for many reasoning-based NLP tasks, suggesting a degree of deductive reasoning capability. However, it remains unclear to which extent LLMs, in both informal and autoformalisation methods, are robust on logical deduction tasks. Moreover, while many LLM-based deduction methods have been proposed, a systematic study that analyses the impact of their design components is lacking. Addressing these two challenges, we propose the first study of the robustness of formal and informal LLM-based deductive reasoning methods. We devise a framework with two families of perturbations: adversarial noise and counterfactual statements, which jointly generate seven perturbed datasets. We organize the landscape of LLM reasoners according to their reasoning format, formalisation syntax, and feedback for error recovery. The results show that adversarial noise affects autoformalisation, while counterfactual statements influence all approaches. Detailed feedback does not improve overall accuracy despite reducing syntax errors, pointing to the challenge of LLM-based methods to self-correct effectively.