# Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs

arXiv: https://arxiv.org/abs/2510.09970 · HTML: https://mlsanity.com/paper/2510.09970/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts](https://mlsanity.com/paper/2209.12711/) (2022)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Logical Fallacy Detection](https://mlsanity.com/paper/2202.13758/) (2022)
- [Faithful Question Answering with Monte-Carlo Planning](https://mlsanity.com/paper/2305.02556/) (2023)
- [Which is better? Exploring Prompting Strategy For LLM-based Metrics](https://mlsanity.com/paper/2311.03754/) (2023)
- [Large Language Models Are Better Logical Fallacy Reasoners with Counterargument, Explanation, and Goal-Aware Prompt Formulation](https://mlsanity.com/paper/2503.23363/) (2025)
- [Boosting Logical Fallacy Reasoning in LLMs via Logical Structure Tree](https://mlsanity.com/paper/2410.12048/) (2024)
- [Conversational AI increases political knowledge as effectively as self-directed internet search](https://mlsanity.com/paper/2509.05219/) (2025)
- [ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering](https://mlsanity.com/paper/2409.11589/) (2024)
- [MAQA: A Multimodal QA Benchmark for Negation](https://mlsanity.com/paper/2301.03238/) (2023)
- [A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoning](https://mlsanity.com/paper/2508.03366/) (2025)
- [When Prolog meets generative models: a new approach for managing knowledge and planning in robotic applications](https://mlsanity.com/paper/2309.15049/) (2023)

## Built on by (most cited first, 1 of 1)

- [ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation](https://mlsanity.com/paper/2606.21867/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) suffer from critical reasoning gaps, including a tendency to hallucinate and poor accuracy in classifying logical fallacies. This limitation stems from their default System 1 processing, which is fast and intuitive, whereas reliable reasoning requires the deliberate, effortful System 2 approach (Kahneman, 2011; Li et al., 2025). Since full System 2 training is often prohibitively expensive, we explore a low-cost, instruction-based intervention to bridge this gap. Our methodology introduces a novel stepwise instruction dataset that decomposes fallacy classification into a series of atomic procedural steps (simple binary questions). We further augment this with a final verification step where models consult a relational knowledge graph of related fallacies. This procedural, rule-based intervention yields a significant improvement in LLM logical fallacy classification. Crucially, the approach also provides enhanced transparency into the LLMs' decision-making, highlighting a practical pathway for Neuro-symbolic architectures to address LLM reasoning deficits.