# Argument Quality Assessment in the Age of Instruction-Following Large Language Models

arXiv: https://arxiv.org/abs/2403.16084 · HTML: https://mlsanity.com/paper/2403.16084/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 137 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis](https://mlsanity.com/paper/1911.11408/) (2019)
- [Automatic Argument Quality Assessment -- New Datasets and Methods](https://mlsanity.com/paper/1909.01007/) (2019)
- [Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender](https://mlsanity.com/paper/2202.11923/) (2022)
- [Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion](https://mlsanity.com/paper/1708.09085/) (2017)
- [Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation](https://mlsanity.com/paper/1802.06613/) (2018)
- [Exploring the Role of Prior Beliefs for Argument Persuasion](https://mlsanity.com/paper/1906.11301/) (2019)
- [Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network](https://mlsanity.com/paper/1907.08971/) (2019)
- [Exploring the Role of Argument Structure in Online Debate Persuasion](https://mlsanity.com/paper/2010.03538/) (2020)
- [A Corpus for Modeling User and Language Effects in Argumentation on Online Debating](https://mlsanity.com/paper/1906.11310/) (2019)
- [The Moral Debater: A Study on the Computational Generation of Morally Framed Arguments](https://mlsanity.com/paper/2203.14563/) (2022)
- [The Role of Pragmatic and Discourse Context in Determining Argument Impact](https://mlsanity.com/paper/2004.03034/) (2020)
- [Detecting Community Sensitive Norm Violations in Online Conversations](https://mlsanity.com/paper/2110.04419/) (2021)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [Finding Convincing Arguments Using Scalable Bayesian Preference Learning](https://mlsanity.com/paper/1806.02418/) (2018)
- [Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing](https://mlsanity.com/paper/2006.00843/) (2020)
- [Scientia Potentia Est -- On the Role of Knowledge in Computational Argumentation](https://mlsanity.com/paper/2107.00281/) (2021)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [Argument Mining Driven Analysis of Peer-Reviews](https://mlsanity.com/paper/2012.07743/) (2020)
- [Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale](https://mlsanity.com/paper/2101.10250/) (2021)
- [Intrinsic Quality Assessment of Arguments](https://mlsanity.com/paper/2010.12473/) (2020)
- [Reason Against the Machine: Future Directions for Mass Online Deliberation](https://mlsanity.com/paper/2107.12711/) (2021)

## Built on by (most cited first, 4 of 4)

- [AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs](https://mlsanity.com/paper/2510.13912/) (2025)
- [Categorical Emotions or Appraisals - Which Emotion Model Explains Argument Convincingness Better?](https://mlsanity.com/paper/2511.07162/) (2025)
- [Argument Reconstruction as Supervision for Critical Thinking in LLMs](https://mlsanity.com/paper/2603.17432/) (2026)
- [TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments](https://mlsanity.com/paper/2607.12480/) (2026)

## Abstract (arXiv)

The computational treatment of arguments on controversial issues has been subject to extensive NLP research, due to its envisioned impact on opinion formation, decision making, writing education, and the like. A critical task in any such application is the assessment of an argument's quality - but it is also particularly challenging. In this position paper, we start from a brief survey of argument quality research, where we identify the diversity of quality notions and the subjectiveness of their perception as the main hurdles towards substantial progress on argument quality assessment. We argue that the capabilities of instruction-following large language models (LLMs) to leverage knowledge across contexts enable a much more reliable assessment. Rather than just fine-tuning LLMs towards leaderboard chasing on assessment tasks, they need to be instructed systematically with argumentation theories and scenarios as well as with ways to solve argument-related problems. We discuss the real-world opportunities and ethical issues emerging thereby.