# Towards a Perspectivist Turn in Argument Quality Assessment

arXiv: https://arxiv.org/abs/2502.14501 · HTML: https://mlsanity.com/paper/2502.14501/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 145 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 40)

- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions](https://mlsanity.com/paper/1602.01103/) (2016)
- [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018)
- [A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis](https://mlsanity.com/paper/1911.11408/) (2019)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [Automatic Argument Quality Assessment -- New Datasets and Methods](https://mlsanity.com/paper/1909.01007/) (2019)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop](https://mlsanity.com/paper/1909.01362/) (2019)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)
- [Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion](https://mlsanity.com/paper/1708.09085/) (2017)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Argotario: Computational Argumentation Meets Serious Games](https://mlsanity.com/paper/1707.06002/) (2017)
- [Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network](https://mlsanity.com/paper/1907.08971/) (2019)
- [DeliData: A dataset for deliberation in multi-party problem solving](https://mlsanity.com/paper/2108.05271/) (2021)
- [Conversational flow in Oxford-style debates](https://mlsanity.com/paper/1604.03114/) (2016)
- [Are Large Language Models Reliable Argument Quality Annotators?](https://mlsanity.com/paper/2404.09696/) (2024)
- [A Corpus for Modeling User and Language Effects in Argumentation on Online Debating](https://mlsanity.com/paper/1906.11310/) (2019)
- [Multitask Instruction-based Prompting for Fallacy Recognition](https://mlsanity.com/paper/2301.09992/) (2023)
- [Multilingual Argument Mining: Datasets and Analysis](https://mlsanity.com/paper/2010.06432/) (2020)
- [The Role of Pragmatic and Discourse Context in Determining Argument Impact](https://mlsanity.com/paper/2004.03034/) (2020)
- [I Beg to Differ: A study of constructive disagreement in online conversations](https://mlsanity.com/paper/2101.10917/) (2021)
- [Annotator-Centric Active Learning for Subjective NLP Tasks](https://mlsanity.com/paper/2404.15720/) (2024)
- [Conversational Markers of Constructive Discussions](https://mlsanity.com/paper/1604.07407/) (2016)
- [Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing](https://mlsanity.com/paper/2006.00843/) (2020)
- [ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation](https://mlsanity.com/paper/2310.14979/) (2023)
- [Detecting Attackable Sentences in Arguments](https://mlsanity.com/paper/2010.02660/) (2020)
- [Missci: Reconstructing Fallacies in Misrepresented Science](https://mlsanity.com/paper/2406.03181/) (2024)
- [Intrinsic Quality Assessment of Arguments](https://mlsanity.com/paper/2010.12473/) (2020)

## Built on by (most cited first, 5 of 5)

- [Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays](https://mlsanity.com/paper/2602.04604/) (2026)
- [From Emotion to Expression: Theoretical Foundations and Resources for Fear Speech](https://mlsanity.com/paper/2601.17132/) (2026)
- [Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach](https://mlsanity.com/paper/2605.28313/) (2026)
- [Persuasion Index: A Theory-Guided Framework for Persuasion Analysis](https://mlsanity.com/paper/2606.14580/) (2026)
- [Challenges in annotations by humans and LLMs: A case study of evaluative language](https://mlsanity.com/paper/2607.28119/) (2026)

## Abstract (arXiv)

The assessment of argument quality depends on well-established logical, rhetorical, and dialectical properties that are unavoidably subjective: multiple valid assessments may exist, there is no unequivocal ground truth. This aligns with recent paths in machine learning, which embrace the co-existence of different perspectives. However, this potential remains largely unexplored in NLP research on argument quality. One crucial reason seems to be the yet unexplored availability of suitable datasets. We fill this gap by conducting a systematic review of argument quality datasets. We assign them to a multi-layered categorization targeting two aspects: (a) What has been annotated: we collect the quality dimensions covered in datasets and consolidate them in an overarching taxonomy, increasing dataset comparability and interoperability. (b) Who annotated: we survey what information is given about annotators, enabling perspectivist research and grounding our recommendations for future actions. To this end, we discuss datasets suitable for developing perspectivist models (i.e., those containing individual, non-aggregated annotations), and we showcase the importance of a controlled selection of annotators in a pilot study.