# Can Language Models Recognize Convincing Arguments?

arXiv: https://arxiv.org/abs/2404.00750 · HTML: https://mlsanity.com/paper/2404.00750/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 31 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions](https://mlsanity.com/paper/1602.01103/) (2016)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Argumentation Mining in User-Generated Web Discourse](https://mlsanity.com/paper/1601.02403/) (2016)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Exploring the Role of Prior Beliefs for Argument Persuasion](https://mlsanity.com/paper/1906.11301/) (2019)
- [Overview of ImageArg-2023: The First Shared Task in Multimodal Argument Mining](https://mlsanity.com/paper/2310.12172/) (2023)

## Built on by (most cited first, 21 of 21)

- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debate](https://mlsanity.com/paper/2406.14711/) (2024)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Training Language Models to Win Debates with Self-Play Improves Judge Accuracy](https://mlsanity.com/paper/2409.16636/) (2024)
- [Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models](https://mlsanity.com/paper/2412.17128/) (2024)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey](https://mlsanity.com/paper/2503.01513/) (2025)
- [Persuasiveness of Generated Free-Text Rationales in Subjective Decisions: A Case Study on Pairwise Argument Ranking](https://mlsanity.com/paper/2406.13905/) (2024)
- [AutoPersuade: A Framework for Evaluating and Explaining Persuasive Arguments](https://mlsanity.com/paper/2410.08917/) (2024)
- [CAP: Evaluation of Persuasive and Creative Image Generation](https://mlsanity.com/paper/2412.10426/) (2024)
- [AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs](https://mlsanity.com/paper/2510.13912/) (2025)
- [Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation](https://mlsanity.com/paper/2506.05062/) (2025)
- [Detecting Winning Arguments with Large Language Models and Persuasion Strategies](https://mlsanity.com/paper/2601.10660/) (2026)
- [Trust Me, I Can Convince You: The Contextualized Argument Appraisal Framework](https://mlsanity.com/paper/2509.17844/) (2025)
- [Working with Large Language Models to Enhance Messaging Effectiveness for Vaccine Confidence](https://mlsanity.com/paper/2504.09857/) (2025)
- [Echoes of Agreement: Argument Driven Opinion Shifts in Large Language Models](https://mlsanity.com/paper/2508.09759/) (2025)
- [A Comparison of Human and ChatGPT Classification Performance on Complex Social Media Data](https://mlsanity.com/paper/2512.00673/) (2025)
- [GRASP: Deterministic argument ranking in interaction graphs](https://mlsanity.com/paper/2605.19141/) (2026)
- [Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach](https://mlsanity.com/paper/2605.28313/) (2026)
- [A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing](https://mlsanity.com/paper/2606.05330/) (2026)

## Abstract (arXiv)

The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans. We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance. The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact. (https://go.epfl.ch/persuasion-llm)