mlsanityData to 14 Oct 2026

Evaluating the Deductive Competence of Large Language Models

arXiv 2309.05452 ↗First posted 11 Sep 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

30 of 56 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
18 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2019How Can We Know What Language Models Know?cited by 959
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  5. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  6. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  7. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  8. 2018A Simple Method for Commonsense Reasoningcited by 223
  9. 2022Language models show human-like content effects on reasoning taskscited by 173
  10. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  11. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  12. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  13. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  14. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  15. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  16. 2022Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTcited by 27
  17. 2019Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignmentcited by 20
  18. 2019Unsupervised Context Rewriting for Open Domain Conversationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 3.

12 papers.

  1. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  2. 2025Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steeringcited by 18
  3. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
  4. 2024Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Modelscited by 8
  5. 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 4
  6. 2024LLM-based Extraction of Contradictions from Patentscited by 3
  7. 2025SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Taskscited by 3
  8. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Modelscited by 2
  9. 2026A Regression Framework for Understanding Prompt Component Impact on LLM Performancecited by 1
  10. 2024Automotive innovation landscaping using LLMcited by 0
  11. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Modelscited by 0
  12. 2026Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  2. 2024The Role of Deductive and Inductive Reasoning in Large Language Models
  3. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
  4. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  5. 2022Towards Reasoning in Large Language Models: A Survey
  6. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  7. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  8. 2025Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?