# Are LLMs Good Zero-Shot Fallacy Classifiers?

arXiv: https://arxiv.org/abs/2410.15050 · HTML: https://mlsanity.com/paper/2410.15050/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 19 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Argotario: Computational Argumentation Meets Serious Games](https://mlsanity.com/paper/1707.06002/) (2017)
- [InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling](https://mlsanity.com/paper/2304.03544/) (2023)
- [Multitask Instruction-based Prompting for Fallacy Recognition](https://mlsanity.com/paper/2301.09992/) (2023)
- [How susceptible are LLMs to Logical Fallacies?](https://mlsanity.com/paper/2308.09853/) (2023)
- [Towards the TopMost: A Topic Modeling System Toolkit](https://mlsanity.com/paper/2309.06908/) (2023)
- [On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling](https://mlsanity.com/paper/2401.14113/) (2024)
- [Autoformalizing Natural Language to First-Order Logic: A Case Study in Logical Fallacy Detection](https://mlsanity.com/paper/2405.02318/) (2024)
- [Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion](https://mlsanity.com/paper/2405.17957/) (2024)
- [Evaluation of an LLM in Identifying Logical Fallacies: A Call for Rigor When Adopting LLMs in HCI Research](https://mlsanity.com/paper/2404.05213/) (2024)
- [Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition](https://mlsanity.com/paper/2311.09552/) (2023)

## Built on by (most cited first, 12 of 12)

- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detection](https://mlsanity.com/paper/2506.08477/) (2025)
- [To trust or not to trust: Attention-based Trust Management for LLM Multi-Agent Systems](https://mlsanity.com/paper/2506.02546/) (2025)
- [Curriculum Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2411.18126/) (2024)
- [CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Base](https://mlsanity.com/paper/2502.12591/) (2025)
- [Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation](https://mlsanity.com/paper/2504.13054/) (2025)
- [SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation](https://mlsanity.com/paper/2505.14419/) (2025)
- [Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification](https://mlsanity.com/paper/2606.26698/) (2026)
- [Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation](https://mlsanity.com/paper/2606.01020/) (2026)
- [Discourse-Aware Policy Analysis with Argumentation: A Hybrid LLM-Symbolic Framework for Disaster Governance](https://mlsanity.com/paper/2607.13260/) (2026)
- [Author Representation Strategies for Zero-Shot Authorship Attribution: A Comparative Study of LLM-Based and Embedding-Based Approaches](https://mlsanity.com/paper/2610.03531/) (2026)
- [UZHCL_for_the_win at Touche: Encoder Fine-Tuning, Synthetic Augmentation, and an Underpowered Encoder-LLM Comparison](https://mlsanity.com/paper/2610.17077/) (2026)

## Abstract (arXiv)

Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities. In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification. To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different task-specific instructions such as extraction, summarization, and Chain-of-Thought reasoning. With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers. In general, LLMs under single-round prompting schemes have achieved acceptable zero-shot performances compared to the best full-shot baselines and can outperform them in all OOD inference scenarios and some open-domain tasks. Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs. Our analysis further underlines the future research on zero-shot fallacy classification. Codes and data are available at: https://github.com/panFJCharlotte98/Fallacy_Detection.