mlsanityData to 7 Oct 2026

A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

arXiv 2311.00445 ↗First posted 1 Nov 2023cs.CL cs.AI cs.LGcited by 20

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises. Psychologists have documented several ways in which humans' inferences deviate from the rules of logic. Do language models, which are trained on text generated by humans, replicate such human biases, or are they able to overcome them? Focusing on the case of syllogisms -- inferences from two simple premises -- we show that, within the PaLM2 family of transformer language models, larger models are more logical than smaller ones, and also more logical than humans. At the same time, even the largest models make systematic errors, some of which mirror human reasoning biases: they show sensitivity to the (irrelevant) ordering of the variables in the syllogism, and draw confident but incorrect inferences from particular syllogisms (syllogistic fallacies). Overall, we find that language models often mimic the human biases included in their training data, but are able to overcome them in some cases.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 65 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

26 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism23 Oct
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  3. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
  4. 2023PaLM 2 Technical Report17 May
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  6. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  7. 2022Language models show human-like content effects on reasoning tasks14 Jul
  8. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoning15 Feb

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Using cognitive psychology to understand GPT-3cited by 389
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
6 more
  1. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  4. 2020Critical Thinking for Language Modelscited by 31
  5. 2021The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generationcited by 15
  6. 2023Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  2. 2023Visual cognition in multimodal large language modelscited by 38
  3. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  4. 2025PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaborationcited by 25
  5. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  6. 2025Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steeringcited by 18
  7. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoningcited by 13
  8. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  9. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  10. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoningcited by 5
  11. 2024Enhancing Conflict Resolution in Language Models via Abstract Argumentationcited by 3
  12. 2025Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worldscited by 2
8 more
  1. 2026Abductive Reasoning with Syllogistic Forms in Large Language Modelscited by 2
  2. 2026Simulating Human Memory with Language Modelscited by 2
  3. 2025From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuitscited by 1
  4. 2026An AI Monkey Gets Grapes for Sure -- Sphere Neural Networks for Reliable Decision-Makingcited by 1
  5. 2026Position: Logical Soundness is not a Reliable Criterion for Neurosymbolic Fact-Checking with LLMscited by 1
  6. 2026Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning -- The Limit of the Scaling Lawcited by 1
  7. 2026Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixescited by 0
  8. 2026Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset
  2. 2023Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases
  3. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
  4. 2023Studying and improving reasoning in humans and machines
  5. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
  6. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
  7. 2025Language Agents Mirror Human Causal Reasoning Biases. How Can We Help Them Think Like Scientists?
  8. 2026Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs