# Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language

arXiv: https://arxiv.org/abs/2406.17753 · HTML: https://mlsanity.com/paper/2406.17753/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions](https://mlsanity.com/paper/1602.01103/) (2016)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles](https://mlsanity.com/paper/2009.02696/) (2020)
- [Automatic Argument Quality Assessment -- New Datasets and Methods](https://mlsanity.com/paper/1909.01007/) (2019)
- [The Persuasive Power of Large Language Models](https://mlsanity.com/paper/2312.15523/) (2023)
- [A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI](https://mlsanity.com/paper/2404.15058/) (2024)
- [Artificial Influence: An Analysis Of AI-Driven Persuasion](https://mlsanity.com/paper/2303.08721/) (2023)
- [Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints](https://mlsanity.com/paper/2302.09185/) (2023)
- [Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network](https://mlsanity.com/paper/1907.08971/) (2019)
- [Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ](https://mlsanity.com/paper/1703.00565/) (2017)

## Built on by (most cited first, 30 of 31)

- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implications](https://mlsanity.com/paper/2411.06837/) (2024)
- [Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models](https://mlsanity.com/paper/2503.01829/) (2025)
- [CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs](https://mlsanity.com/paper/2505.11413/) (2025)
- [Literature Meets Data: A Synergistic Approach to Hypothesis Generation](https://mlsanity.com/paper/2410.17309/) (2024)
- [It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics](https://mlsanity.com/paper/2506.02873/) (2025)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind](https://mlsanity.com/paper/2505.22961/) (2025)
- [How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversations](https://mlsanity.com/paper/2508.05625/) (2025)
- ["I understand your perspective": LLM Persuasion through the Lens of Communicative Action Theory](https://mlsanity.com/paper/2606.08076/) (2026)
- [Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection](https://mlsanity.com/paper/2510.09695/) (2025)
- [Understanding Persuasive Interactions between Generative Social Agents and Humans: The Knowledge-based Persuasion Model (KPM)](https://mlsanity.com/paper/2602.11483/) (2026)
- [Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated](https://mlsanity.com/paper/2410.03723/) (2024)
- [Tailored untruths: How personalisation challenges LLM safeguards](https://mlsanity.com/paper/2510.12993/) (2025)
- [Learning When to Quit in Sales Conversations](https://mlsanity.com/paper/2511.01181/) (2025)
- [The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues](https://mlsanity.com/paper/2603.20907/) (2026)
- [Seeing is Believing? Evaluating Vision-Language Model Susceptibility in Agent-to-Agent Multimodal Persuasion](https://mlsanity.com/paper/2510.22768/) (2025)
- [Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences](https://mlsanity.com/paper/2601.04925/) (2026)
- [When Agents Persuade: Rhetoric Generation and Mitigation in LLMs](https://mlsanity.com/paper/2603.04636/) (2026)
- [AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models](https://mlsanity.com/paper/2506.11110/) (2025)
- [Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models](https://mlsanity.com/paper/2601.17082/) (2026)
- [Communicate-Predict-Act: Evaluating Social Intelligence of Agents](https://mlsanity.com/paper/2604.08727/) (2026)
- [Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations](https://mlsanity.com/paper/2604.22109/) (2026)
- [Human Decision-Making with Persuasive and Narrative LLM Explanations](https://mlsanity.com/paper/2605.23867/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)
- [A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing](https://mlsanity.com/paper/2606.05330/) (2026)
- [Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models](https://mlsanity.com/paper/2606.07877/) (2026)
- [Pareto-Guided Teacher Alignment for Fair Personalized Text Generation](https://mlsanity.com/paper/2606.10126/) (2026)
- [Janus: A Benchmark for Goal-Conditioned Information Distortion in LLMs](https://mlsanity.com/paper/2606.10852/) (2026)
- [It Matters How You Say It: Exploring Rhetorical Patterns for AI-Assisted Information Evaluation](https://mlsanity.com/paper/2607.17627/) (2026)

## Abstract (arXiv)

We are exposed to much information trying to influence us, such as teaser messages, debates, politically framed news, and propaganda - all of which use persuasive language. With the recent interest in Large Language Models (LLMs), we study the ability of LLMs to produce persuasive text. As opposed to prior work which focuses on particular domains or types of persuasion, we conduct a general study across various domains to measure and benchmark to what degree LLMs produce persuasive language - both when explicitly instructed to rewrite text to be more or less persuasive and when only instructed to paraphrase. We construct the new dataset Persuasive-Pairs of pairs of a short text and its rewrite by an LLM to amplify or diminish persuasive language. We multi-annotate the pairs on a relative scale for persuasive language: a valuable resource in itself, and for training a regression model to score and benchmark persuasive language, including for new LLMs across domains. In our analysis, we find that different 'personas' in LLaMA3's system prompt change persuasive language substantially, even when only instructed to paraphrase.