# MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification

arXiv: https://arxiv.org/abs/2311.09761 · HTML: https://mlsanity.com/paper/2311.09761/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 61 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Inverse scaling can become U-shaped](https://mlsanity.com/paper/2211.02011/) (2022)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://mlsanity.com/paper/1708.01425/) (2017)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Argotario: Computational Argumentation Meets Serious Games](https://mlsanity.com/paper/1707.06002/) (2017)
- [Multitask Instruction-based Prompting for Fallacy Recognition](https://mlsanity.com/paper/2301.09992/) (2023)
- [How susceptible are LLMs to Logical Fallacies?](https://mlsanity.com/paper/2308.09853/) (2023)
- [#DebateNight: The Role and Influence of Socialbots on Twitter During the 1st 2016 U.S. Presidential Debate](https://mlsanity.com/paper/1802.09808/) (2018)
- [Riposte! A Large Corpus of Counter-Arguments](https://mlsanity.com/paper/1910.03246/) (2019)

## Built on by (most cited first, 4 of 4)

- [Critical Questions Generation: Motivation and Challenges](https://mlsanity.com/paper/2410.14335/) (2024)
- [AMELIA: A Family of Multi-task End-to-end Language Models for Argumentation](https://mlsanity.com/paper/2508.17926/) (2025)
- [Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions](https://mlsanity.com/paper/2509.25539/) (2025)
- [ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety](https://mlsanity.com/paper/2508.20468/) (2025)

## Abstract (arXiv)

We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies.