# Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

arXiv: https://arxiv.org/abs/2402.14856 · HTML: https://mlsanity.com/paper/2402.14856/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 54 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models](https://mlsanity.com/paper/2402.04614/) (2024)
- [Logical Reasoning over Natural Language as Knowledge Representation: A Survey](https://mlsanity.com/paper/2303.12023/) (2023)
- [Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption](https://mlsanity.com/paper/2304.01358/) (2023)
- [A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models](https://mlsanity.com/paper/2311.00445/) (2023)

## Built on by (most cited first, 13 of 13)

- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey](https://mlsanity.com/paper/2404.01869/) (2024)
- [Improving Reasoning Performance in Large Language Models via Representation Engineering](https://mlsanity.com/paper/2504.19483/) (2025)
- [RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues](https://mlsanity.com/paper/2409.12558/) (2024)
- [Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering](https://mlsanity.com/paper/2505.12189/) (2025)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?](https://mlsanity.com/paper/2507.11423/) (2025)
- [Position: Logical Soundness is not a Reliable Criterion for Neurosymbolic Fact-Checking with LLMs](https://mlsanity.com/paper/2604.04177/) (2026)
- [How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models](https://mlsanity.com/paper/2604.15873/) (2026)
- [Boosting Process-Correct CoT Reasoning by Modeling Solvability of Multiple-Choice QA](https://mlsanity.com/paper/2509.25941/) (2025)
- [If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models](https://mlsanity.com/paper/2510.08388/) (2025)
- [LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning](https://mlsanity.com/paper/2608.12626/) (2026)

## Abstract (arXiv)

Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like textitsupposition following or textitchain construction. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.