mlsanityData to 14 Oct 2026

Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning

arXiv 2506.11128 ↗First posted 10 Jun 2025cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We study logical reasoning in language models by asking whether their errors follow established human fallacy patterns. Using the Erotetic Theory of Reasoning (ETR) and its open-source implementation, PyETR, we programmatically generate 383 formally specified reasoning problems and evaluate 38 models. For each response, we judge logical correctness and, when incorrect, whether it matches an ETR-predicted fallacy. Two results stand out: (i) as a capability proxy (Chatbot Arena Elo) increases, a larger share of a model's incorrect answers are ETR-predicted fallacies (ρ=0.360, p=0.0265), while overall correctness on this dataset shows no correlation with capability; (ii) reversing premise order significantly reduces fallacy production for many models, mirroring human order effects. Methodologically, PyETR provides an open-source pipeline for unbounded, synthetic, contamination-resistant reasoning tests linked to a cognitive theory, enabling analyses that focus on error composition rather than error rate.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 46 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

12 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models16 Jan

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  8. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  9. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 32
  10. 2023Humans in Humans Out: On GPT Converging Toward Common Sense in both Success and Failurecited by 10
  11. 2021Measurement as governance in and for responsible AIcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026NeuReasoner: Theory-grounded Mapping of Reasoning Elicitation Boundariescited by 2
  2. 2026Half-Truths Break Similarity-Based Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Conditional and Modal Reasoning in Large Language Models
  2. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
  3. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset
  4. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding
  5. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
  6. 2022Language models show human-like content effects on reasoning tasks
  7. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
  8. 2025Towards Evaluting Fake Reasoning Bias in Language Models