# How susceptible are LLMs to Logical Fallacies?

arXiv: https://arxiv.org/abs/2308.09853 · HTML: https://mlsanity.com/paper/2308.09853/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [The Limitations of Stylometry for Detecting Machine-Generated Fake News](https://mlsanity.com/paper/1908.09805/) (2019)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://mlsanity.com/paper/1708.01425/) (2017)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications](https://mlsanity.com/paper/2307.09162/) (2023)
- [Logical Fallacy Detection](https://mlsanity.com/paper/2202.13758/) (2022)

## Built on by (most cited first, 23 of 23)

- [The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models](https://mlsanity.com/paper/2401.05618/) (2024)
- [VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models](https://mlsanity.com/paper/2404.00210/) (2024)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)
- [Self-Reflection in LLM Agents: Effects on Problem-Solving Performance](https://mlsanity.com/paper/2405.06682/) (2024)
- [Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs](https://mlsanity.com/paper/2404.17833/) (2024)
- [Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD](https://mlsanity.com/paper/2508.17450/) (2025)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Are LLMs Good Zero-Shot Fallacy Classifiers?](https://mlsanity.com/paper/2410.15050/) (2024)
- [Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment](https://mlsanity.com/paper/2411.11731/) (2024)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition](https://mlsanity.com/paper/2311.09552/) (2023)
- [MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification](https://mlsanity.com/paper/2311.09761/) (2023)
- [Attacks by Content: Automated Fact-checking is an AI Security Issue](https://mlsanity.com/paper/2510.11238/) (2025)
- [Critical Questions Generation: Motivation and Challenges](https://mlsanity.com/paper/2410.14335/) (2024)
- [Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection](https://mlsanity.com/paper/2510.09695/) (2025)
- [From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models](https://mlsanity.com/paper/2312.17256/) (2023)
- [Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs](https://mlsanity.com/paper/2605.29928/) (2026)
- [Can a Hallucinating Model help in Reducing Human "Hallucination"?](https://mlsanity.com/paper/2405.00843/) (2024)
- [Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation](https://mlsanity.com/paper/2606.01020/) (2026)
- [Collaborative Disagreement Resolution for Scalable Oversight](https://mlsanity.com/paper/2607.01251/) (2026)
- [DCGC: Draft-Conditioned Global Correction for Complex Reasoning with Masked Diffusion Models](https://mlsanity.com/paper/2608.25428/) (2026)
- [From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interaction](https://mlsanity.com/paper/2609.07573/) (2026)
- [Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks](https://mlsanity.com/paper/2609.28673/) (2026)

## Abstract (arXiv)

This paper investigates the rational thinking capability of Large Language Models (LLMs) in multi-round argumentative debates by exploring the impact of fallacious arguments on their logical reasoning performance. More specifically, we present Logic Competence Measurement Benchmark (LOGICOM), a diagnostic benchmark to assess the robustness of LLMs against logical fallacies. LOGICOM involves two agents: a persuader and a debater engaging in a multi-round debate on a controversial topic, where the persuader tries to convince the debater of the correctness of its claim. First, LOGICOM assesses the potential of LLMs to change their opinions through reasoning. Then, it evaluates the debater's performance in logical reasoning by contrasting the scenario where the persuader employs logical fallacies against one where logical reasoning is used. We use this benchmark to evaluate the performance of GPT-3.5 and GPT-4 using a dataset containing controversial topics, claims, and reasons supporting them. Our findings indicate that both GPT-3.5 and GPT-4 can adjust their opinion through reasoning. However, when presented with logical fallacies, GPT-3.5 and GPT-4 are erroneously convinced 41% and 69% more often, respectively, compared to when logical reasoning is used. Finally, we introduce a new dataset containing over 5k pairs of logical vs. fallacious arguments. The source code and dataset of this work are made publicly available.