mlsanityData to 14 Oct 2026

Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

arXiv 2306.12567 ↗First posted 21 Jun 2023cs.CL cs.AIcited by 14

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper investigates whether current large language models exhibit biases in logical reasoning, similar to humans. Specifically, we focus on syllogistic reasoning, a well-studied form of inference in the cognitive science of human deduction. To facilitate our analysis, we introduce a dataset called NeuBAROCO, originally designed for psychological experiments that assess human logical abilities in syllogistic reasoning. The dataset consists of syllogistic inferences in both English and Japanese. We examine three types of biases observed in human syllogistic reasoning: belief biases, conversion errors, and atmosphere effects. Our findings demonstrate that current large language models struggle more with problems involving these three types of biases.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 31 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

9 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Can Transformers Reason in Fragments of Natural Language?10 Nov
  2. 2022Language models show human-like content effects on reasoning tasks14 Jul

Other references · 7

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
  7. 2019Can neural networks understand monotonicity reasoning?cited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 1; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  2. 2024Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processescited by 39
  3. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  4. 2026Large Language Model Reasoning Failurescited by 21
  5. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  6. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Datasetcited by 14
  7. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  8. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  9. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  10. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Modelscited by 2
  11. 2026Abductive Reasoning with Syllogistic Forms in Large Language Modelscited by 2
  12. 2025From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuitscited by 1
2 more
  1. 2026A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Modelscited by 0
  2. 2026Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
  2. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
  3. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
  4. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
  5. 2024(Ir)rationality and Cognitive Biases in Large Language Models
  6. 2025Towards Evaluting Fake Reasoning Bias in Language Models
  7. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
  8. 2024Conditional and Modal Reasoning in Large Language Models