# A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis

arXiv: https://arxiv.org/abs/1911.11408 · HTML: https://mlsanity.com/paper/1911.11408/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 30 of 67)

- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning](https://mlsanity.com/paper/2104.07644/) (2021)
- [Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements](https://mlsanity.com/paper/2402.10614/) (2024)
- [The Touch'e23-ValueEval Dataset for Identifying Human Values behind Arguments](https://mlsanity.com/paper/2301.13771/) (2023)
- [Overview of the 2021 Key Point Analysis Shared Task](https://mlsanity.com/paper/2110.10577/) (2021)
- [IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks](https://mlsanity.com/paper/2203.12257/) (2022)
- [Every Bite Is an Experience: Key Point Analysis of Business Reviews](https://mlsanity.com/paper/2106.06758/) (2021)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals](https://mlsanity.com/paper/2311.03998/) (2023)
- [Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation](https://mlsanity.com/paper/2107.00281/) (2021)
- [Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models](https://mlsanity.com/paper/2412.17128/) (2024)
- [Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale](https://mlsanity.com/paper/2101.10250/) (2021)
- [Large Language Models in Argument Mining: A Survey](https://mlsanity.com/paper/2506.16383/) (2025)
- [Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation](https://mlsanity.com/paper/2406.03151/) (2024)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [Team Enigma at ArgMining-EMNLP 2021: Leveraging Pre-trained Language Models for Key Point Matching](https://mlsanity.com/paper/2110.12370/) (2021)
- [Assessing the Sufficiency of Arguments through Conclusion Generation](https://mlsanity.com/paper/2110.13495/) (2021)
- [Project Debater APIs: Decomposing the AI Grand Challenge](https://mlsanity.com/paper/2110.01029/) (2021)
- [To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support](https://mlsanity.com/paper/2305.16799/) (2023)
- [Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text](https://mlsanity.com/paper/2409.02078/) (2024)
- [DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models](https://mlsanity.com/paper/2110.01509/) (2021)
- [Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Prediction](https://mlsanity.com/paper/2209.08966/) (2022)
- [Persuasiveness of Generated Free-Text Rationales in Subjective Decisions: A Case Study on Pairwise Argument Ranking](https://mlsanity.com/paper/2406.13905/) (2024)
- [Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models](https://mlsanity.com/paper/2305.08625/) (2023)
- [A Hybrid Intelligence Method for Argument Mining](https://mlsanity.com/paper/2403.09713/) (2024)
- [Prompted Aspect Key Point Analysis for Quantitative Review Summarization](https://mlsanity.com/paper/2407.14049/) (2024)
- [From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls](https://mlsanity.com/paper/2410.01039/) (2024)
- [An Item Response Theory Framework for Persuasion](https://mlsanity.com/paper/2204.11337/) (2022)
- [Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!](https://mlsanity.com/paper/2311.03153/) (2023)
- [Towards a Perspectivist Turn in Argument Quality Assessment](https://mlsanity.com/paper/2502.14501/) (2025)

## Abstract (arXiv)

Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.