# An LLM Feature-based Framework for Dialogue Constructiveness Assessment

arXiv: https://arxiv.org/abs/2406.14760 · HTML: https://mlsanity.com/paper/2406.14760/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions](https://mlsanity.com/paper/1602.01103/) (2016)
- [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning](https://mlsanity.com/paper/2305.17256/) (2023)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies](https://mlsanity.com/paper/2001.04564/) (2020)
- [Nested cross-validation when selecting classifiers is overzealous for most practical applications](https://mlsanity.com/paper/1809.09446/) (2018)
- [I Beg to Differ: A study of constructive disagreement in online conversations](https://mlsanity.com/paper/2101.10917/) (2021)
- [Conversational Markers of Constructive Discussions](https://mlsanity.com/paper/1604.07407/) (2016)
- [How to disagree well: Investigating the dispute tactics used on Wikipedia](https://mlsanity.com/paper/2212.08353/) (2022)
- [Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models](https://mlsanity.com/paper/2405.14646/) (2024)

## Built on by (most cited first, 9 of 9)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [AutoQual: An LLM Agent for Automated Discovery of Interpretable Features for Review Quality Assessment](https://mlsanity.com/paper/2510.08081/) (2025)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [Improving Medical Communication using Rubric-Guided Counterfactual Recommendations](https://mlsanity.com/paper/2606.18889/) (2026)
- [From Divergence to Consensus: Evaluating the Role of Large Language Models in Facilitating Agreement through Adaptive Strategies](https://mlsanity.com/paper/2503.15521/) (2025)
- [Computational Analysis of Conversation Dynamics through Participant Responsivity](https://mlsanity.com/paper/2509.16464/) (2025)
- [Measuring Semantic Progress in Multi-turn Dialogue via Information Gain](https://mlsanity.com/paper/2606.12332/) (2026)
- [Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions](https://mlsanity.com/paper/2609.34326/) (2026)

## Abstract (arXiv)

Research on dialogue constructiveness assessment focuses on (i) analysing conversational factors that influence individuals to take specific actions, win debates, change their perspectives or broaden their open-mindedness and (ii) predicting constructiveness outcomes following dialogues for such use cases. These objectives can be achieved by training either interpretable feature-based models (which often involve costly human annotations) or neural models such as pre-trained language models (which have empirically shown higher task accuracy but lack interpretability). In this paper we propose an LLM feature-based framework for dialogue constructiveness assessment that combines the strengths of feature-based and neural approaches, while mitigating their downsides. The framework first defines a set of dataset-independent and interpretable linguistic features, which can be extracted by both prompting an LLM and simple heuristics. Such features are then used to train LLM feature-based models. We apply this framework to three datasets of dialogue constructiveness and find that our LLM feature-based models outperform or performs at least as well as standard feature-based models and neural models. We also find that the LLM feature-based model learns more robust prediction rules instead of relying on superficial shortcuts, which often trouble neural models.