mlsanityData to 14 Oct 2026

How susceptible are LLMs to Logical Fallacies?

arXiv 2308.09853 ↗First posted 18 Aug 2023cs.CLcited by 23

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper investigates the rational thinking capability of Large Language Models (LLMs) in multi-round argumentative debates by exploring the impact of fallacious arguments on their logical reasoning performance. More specifically, we present Logic Competence Measurement Benchmark (LOGICOM), a diagnostic benchmark to assess the robustness of LLMs against logical fallacies. LOGICOM involves two agents: a persuader and a debater engaging in a multi-round debate on a controversial topic, where the persuader tries to convince the debater of the correctness of its claim. First, LOGICOM assesses the potential of LLMs to change their opinions through reasoning. Then, it evaluates the debater's performance in logical reasoning by contrasting the scenario where the persuader employs logical fallacies against one where logical reasoning is used. We use this benchmark to evaluate the performance of GPT-3.5 and GPT-4 using a dataset containing controversial topics, claims, and reasons supporting them. Our findings indicate that both GPT-3.5 and GPT-4 can adjust their opinion through reasoning. However, when presented with logical fallacies, GPT-3.5 and GPT-4 are erroneously convinced 41% and 69% more often, respectively, compared to when logical reasoning is used. Finally, we introduce a new dataset containing over 5k pairs of logical vs. fallacious arguments. The source code and dataset of this work are made publicly available.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 25 of its 39 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

25 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond16 Jun
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance26 May
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debate23 May
  4. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  5. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  6. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  7. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Other references · 18

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2019Defending Against Neural Fake Newscited by 559
  10. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  11. 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 209
  12. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
6 more
  1. 2019The Limitations of Stylometry for Detecting Machine-Generated Fake Newscited by 63
  2. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  3. 2017The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrantscited by 56
  4. 2018Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentationcited by 33
  5. 2023Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implicationscited by 29
  6. 2022Logical Fallacy Detectioncited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 4; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 63
  2. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
  3. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  4. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 39
  5. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  6. 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
  7. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  8. 2024Are LLMs Good Zero-Shot Fallacy Classifiers?cited by 12
  9. 2024Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignmentcited by 10
  10. 2025MultiHoax: A Dataset of Multi-hop False-Premise Questionscited by 9
  11. 2023Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognitioncited by 5
  12. 2023MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classificationcited by 4
11 more
  1. 2025Attacks by Content: Automated Fact-checking is an AI Security Issuecited by 4
  2. 2024Critical Questions Generation: Motivation and Challengescited by 3
  3. 2025Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detectioncited by 3
  4. 2023From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Modelscited by 1
  5. 2026Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMscited by 1
  6. 2024Can a Hallucinating Model help in Reducing Human "Hallucination"?cited by 0
  7. 2026Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentationcited by 0
  8. 2026Collaborative Disagreement Resolution for Scalable Oversightcited by 0
  9. 2026DCGC: Draft-Conditioned Global Correction for Complex Reasoning with Masked Diffusion Modelscited by 0
  10. 2026From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interactioncited by 0
  11. 2026Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  2. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
  3. 2024Lost in the Logic: An Evaluation of Large Language Models' Reasoning Capabilities on LSAT Logic Games
  4. 2025Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation
  5. 2025When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)
  6. 2025Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs
  7. 2024Debating with More Persuasive LLMs Leads to More Truthful Answers
  8. 2026How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality