# AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs

arXiv: https://arxiv.org/abs/2404.02761 · HTML: https://mlsanity.com/paper/2404.02761/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 53 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning](https://mlsanity.com/paper/2311.11077/) (2023)

## Built on by (most cited first, 5 of 5)

- [Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers](https://mlsanity.com/paper/2410.22937/) (2024)
- [The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse](https://mlsanity.com/paper/2608.10186/) (2026)
- [Supporting Online Discussions: Integrating AI Into the adhocracy+ Participation Platform To Enhance Deliberation](https://mlsanity.com/paper/2409.07780/) (2024)
- [Incivility and Rigidity: Evaluating the Risks of Fine-Tuning LLMs for Political Argumentation](https://mlsanity.com/paper/2411.16813/) (2024)
- [OET: Optimization-based prompt injection Evaluation Toolkit](https://mlsanity.com/paper/2505.00843/) (2025)

## Abstract (arXiv)

Measuring the quality of contributions in political online discussions is crucial in deliberation research and computer science. Research has identified various indicators to assess online discussion quality, and with deep learning advancements, automating these measures has become feasible. While some studies focus on analyzing specific quality indicators, a comprehensive quality score incorporating various deliberative aspects is often preferred. In this work, we introduce AQuA, an additive score that calculates a unified deliberative quality score from multiple indices for each discussion post. Unlike other singular scores, AQuA preserves information on the deliberative aspects present in comments, enhancing model transparency. We develop adapter models for 20 deliberative indices, and calculate correlation coefficients between experts' annotations and the perceived deliberativeness by non-experts to weigh the individual indices into a single deliberative score. We demonstrate that the AQuA score can be computed easily from pre-trained adapters and aligns well with annotations on other datasets that have not be seen during training. The analysis of experts' vs. non-experts' annotations confirms theoretical findings in the social science literature.