# Fine-grained Fallacy Detection with Human Label Variation

arXiv: https://arxiv.org/abs/2502.13853 · HTML: https://mlsanity.com/paper/2502.13853/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 68 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 11 of 11)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles](https://mlsanity.com/paper/2009.02696/) (2020)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Argotario: Computational Argumentation Meets Serious Games](https://mlsanity.com/paper/1707.06002/) (2017)
- [Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection](https://mlsanity.com/paper/1910.09982/) (2019)
- [Multitask Instruction-based Prompting for Fallacy Recognition](https://mlsanity.com/paper/2301.09992/) (2023)
- [Variationist: Exploring Multifaceted Variation and Bias in Written Language Data](https://mlsanity.com/paper/2406.17647/) (2024)

## Built on by (most cited first, 6 of 6)

- [LLMs as Span Annotators: A Comparative Study of LLMs and Humans](https://mlsanity.com/paper/2504.08697/) (2025)
- [Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection](https://mlsanity.com/paper/2510.09695/) (2025)
- [Strategies for Span Labeling with Large Language Models](https://mlsanity.com/paper/2601.16946/) (2026)
- [ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety](https://mlsanity.com/paper/2508.20468/) (2025)
- [CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents](https://mlsanity.com/paper/2603.03884/) (2026)
- [How Ethos and Pathos Appeals Resonate in Reader Interpretations of Social Media Messages](https://mlsanity.com/paper/2607.00873/) (2026)

## Abstract (arXiv)

We introduce Faina, the first dataset for fallacy detection that embraces multiple plausible answers and natural disagreement. Faina includes over 11K span-level annotations with overlaps across 20 fallacy types on social media posts in Italian about migration, climate change, and public health given by two expert annotators. Through an extensive annotation study that allowed discussion over multiple rounds, we minimize annotation errors whilst keeping signals of human label variation. Moreover, we devise a framework that goes beyond "single ground truth" evaluation and simultaneously accounts for multiple (equally reliable) test sets and the peculiarities of the task, i.e., partial span matches, overlaps, and the varying severity of labeling errors. Our experiments across four fallacy detection setups show that multi-task and multi-label transformer-based approaches are strong baselines across all settings. We release our data, code, and annotation guidelines to foster research on fallacy detection and human label variation more broadly.