# Measuring and Improving Persuasiveness of Large Language Models

arXiv: https://arxiv.org/abs/2410.02653 · HTML: https://mlsanity.com/paper/2410.02653/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 83 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion](https://mlsanity.com/paper/1708.09085/) (2017)
- [Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning](https://mlsanity.com/paper/2203.06875/) (2022)
- [The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter](https://mlsanity.com/paper/1405.1438/) (2014)
- [How to Ask for a Favor: A Case Study on the Success of Altruistic Requests](https://mlsanity.com/paper/1405.3282/) (2014)
- [A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot](https://mlsanity.com/paper/2305.09758/) (2023)
- [Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior](https://mlsanity.com/paper/2309.00359/) (2023)
- [Long-Term Ad Memorability: Understanding & Generating Memorable Ads](https://mlsanity.com/paper/2309.00378/) (2023)
- [Audience-Centric Natural Language Generation via Style Infusion](https://mlsanity.com/paper/2301.10283/) (2023)
- [When do Words Matter? Understanding the Impact of Lexical Choice on Audience Perception using Individual Treatment Effect Estimation](https://mlsanity.com/paper/1811.04890/) (2018)
- [The Untold Story of the Clones: Content-agnostic Factors that Impact YouTube Video Popularity](https://mlsanity.com/paper/1311.6526/) (2013)
- [Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings](https://mlsanity.com/paper/2110.02030/) (2021)
- [Behavior Optimized Image Generation](https://mlsanity.com/paper/2311.10995/) (2023)

## Built on by (most cited first, 20 of 20)

- [Authenticated Delegation and Authorized AI Agents](https://mlsanity.com/paper/2501.09674/) (2025)
- [Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models](https://mlsanity.com/paper/2503.01829/) (2025)
- [It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics](https://mlsanity.com/paper/2506.02873/) (2025)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind](https://mlsanity.com/paper/2505.22961/) (2025)
- [Towards Strategic Persuasion with Language Models](https://mlsanity.com/paper/2509.22989/) (2025)
- [AI Realtor: Towards Grounded Persuasive Language Generation for Automated Copywriting](https://mlsanity.com/paper/2502.16810/) (2025)
- [CAP: Evaluation of Persuasive and Creative Image Generation](https://mlsanity.com/paper/2412.10426/) (2024)
- [Societal Impacts Research Requires Benchmarks for Creative Composition Tasks](https://mlsanity.com/paper/2504.06549/) (2025)
- [Accelerating Social Science Research via Agentic Hypothesization and Experimentation](https://mlsanity.com/paper/2602.07983/) (2026)
- [Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety](https://mlsanity.com/paper/2605.22643/) (2026)
- [Unraveling SITT: Social Influence Technique Taxonomy and Detection with LLMs](https://mlsanity.com/paper/2506.00061/) (2025)
- [The Hunger Game Debate: On the Emergence of Over-Competition in Multi-Agent Systems](https://mlsanity.com/paper/2509.26126/) (2025)
- [Verification Required: The Impact of Information Credibility on AI Persuasion](https://mlsanity.com/paper/2602.00970/) (2026)
- [Argument Reconstruction as Supervision for Critical Thinking in LLMs](https://mlsanity.com/paper/2603.17432/) (2026)
- [Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations](https://mlsanity.com/paper/2604.22109/) (2026)
- [Sense it with your eyes: Sensation Generation and Understanding for Advertisements](https://mlsanity.com/paper/2607.25314/) (2026)
- [ARGUS: Theory-of-Mind Guided Argument Generation with Strategy-Aware Planning and Knowledge Grounding](https://mlsanity.com/paper/2608.20405/) (2026)
- [Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization](https://mlsanity.com/paper/2610.09964/) (2026)
- [Understanding and Reducing Misalignment in Persuasion Optimization](https://mlsanity.com/paper/2610.16070/) (2026)

## Abstract (arXiv)

LLMs are increasingly being used in workflows involving generating content to be consumed by humans (e.g., marketing) and also in directly interacting with humans (e.g., through chatbots). The development of such systems that are capable of generating verifiably persuasive messages presents both opportunities and challenges for society. On the one hand, such systems could positively impact domains like advertising and social good, such as addressing drug addiction, and on the other, they could be misused for spreading misinformation and shaping political opinions. To channel LLMs' impact on society, we need to develop systems to measure and benchmark their persuasiveness. With this motivation, we introduce PersuasionBench and PersuasionArena, the first large-scale benchmark and arena containing a battery of tasks to measure the persuasion ability of generative models automatically. We investigate to what extent LLMs know and leverage linguistic patterns that can help them generate more persuasive language. Our findings indicate that the persuasiveness of LLMs correlates positively with model size, but smaller models can also be made to have a higher persuasiveness than much larger models. Notably, targeted training using synthetic and natural datasets significantly enhances smaller models' persuasive capabilities, challenging scale-dependent assumptions. Our findings carry key implications for both model developers and policymakers. For instance, while the EU AI Act and California's SB-1047 aim to regulate AI models based on the number of floating point operations, we demonstrate that simple metrics like this alone fail to capture the full scope of AI's societal impact. We invite the community to explore and contribute to PersuasionArena and PersuasionBench, available at https://bit.ly/measure-persuasion, to advance our understanding of AI-driven persuasion and its societal implications.