# Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models

arXiv: https://arxiv.org/abs/2503.01829 · HTML: https://mlsanity.com/paper/2503.01829/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis](https://mlsanity.com/paper/2402.05863/) (2024)
- [Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims](https://mlsanity.com/paper/1906.03538/) (2019)
- [The Persuasive Power of Large Language Models](https://mlsanity.com/paper/2312.15523/) (2023)
- [Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language](https://mlsanity.com/paper/2406.17753/) (2024)
- [Measuring and Improving Persuasiveness of Large Language Models](https://mlsanity.com/paper/2410.02653/) (2024)
- [Teaching Models to Balance Resisting and Accepting Persuasion](https://mlsanity.com/paper/2410.14596/) (2024)
- [Zero-shot Persuasive Chatbots with LLM-Generated Strategies and Information Retrieval](https://mlsanity.com/paper/2407.03585/) (2024)

## Built on by (most cited first, 27 of 27)

- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning](https://mlsanity.com/paper/2510.07517/) (2025)
- [It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics](https://mlsanity.com/paper/2506.02873/) (2025)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind](https://mlsanity.com/paper/2505.22961/) (2025)
- [Towards Strategic Persuasion with Language Models](https://mlsanity.com/paper/2509.22989/) (2025)
- [Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Risks](https://mlsanity.com/paper/2505.19806/) (2025)
- [Scheming Ability in LLM-to-LLM Strategic Interactions](https://mlsanity.com/paper/2510.12826/) (2025)
- [Sell More, Play Less: Benchmarking LLM Realistic Selling Skill](https://mlsanity.com/paper/2604.07054/) (2026)
- [Social Dynamics as Critical Vulnerabilities that Undermine Objective Decision-Making in LLM Collectives](https://mlsanity.com/paper/2604.06091/) (2026)
- [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026)
- [Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs](https://mlsanity.com/paper/2608.11624/) (2026)
- [Seeing is Believing? Evaluating Vision-Language Model Susceptibility in Agent-to-Agent Multimodal Persuasion](https://mlsanity.com/paper/2510.22768/) (2025)
- [Persuasion Propagation: Does Persuasion Change AI Agent Behavior?](https://mlsanity.com/paper/2602.00851/) (2026)
- [AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models](https://mlsanity.com/paper/2602.16639/) (2026)
- [Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies](https://mlsanity.com/paper/2602.22696/) (2026)
- [Is Monitoring Enough? Strategic Agent Selection For Stealthy Attack in Multi-Agent Discussions](https://mlsanity.com/paper/2603.21194/) (2026)
- [Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Tools](https://mlsanity.com/paper/2604.26233/) (2026)
- [LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight](https://mlsanity.com/paper/2605.08321/) (2026)
- [Learning with Conflicts of Interest](https://mlsanity.com/paper/2605.15504/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)
- [A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing](https://mlsanity.com/paper/2606.05330/) (2026)
- [Contextualized Counterspeech Can Be More Persuasive Than Generic Counterspeech](https://mlsanity.com/paper/2607.26236/) (2026)
- [CC-Mediation: Evaluating Large Language Models for Cross-Cultural Conflict Mediation](https://mlsanity.com/paper/2609.04855/) (2026)
- [A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement](https://mlsanity.com/paper/2609.14178/) (2026)
- [Peer Influence across Heterogeneous AI Models](https://mlsanity.com/paper/2610.03095/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) demonstrate persuasive capabilities that rival human-level persuasion. While these capabilities can be used for social good, they also present risks of potential misuse. Beyond the concern of how LLMs persuade others, their own susceptibility to persuasion poses a critical alignment challenge, raising questions about robustness, safety, and adherence to ethical principles. To study these dynamics, we introduce Persuade Me If You Can (PMIYC), an automated framework for evaluating persuasiveness and susceptibility to persuasion in multi-agent interactions. Our framework offers a scalable alternative to the costly and time-intensive human annotation process typically used to study persuasion in LLMs. PMIYC automatically conducts multi-turn conversations between Persuader and Persuadee agents, measuring both the effectiveness of and susceptibility to persuasion. Our comprehensive evaluation spans a diverse set of LLMs and persuasion settings (e.g., subjective and misinformation scenarios). We validate the efficacy of our framework through human evaluations and demonstrate alignment with human assessments from prior studies. Through PMIYC, we find that Llama-3.3-70B and GPT-4o exhibit similar persuasive effectiveness, outperforming Claude 3 Haiku by 30%. However, GPT-4o demonstrates over 50% greater resistance to persuasion for misinformation compared to Llama-3.3-70B. Notably, o4-mini emerges as both an effective persuader, and a resistant persuadee. These findings provide empirical insights into the persuasive dynamics of LLMs and contribute to the development of safer AI systems.