# Evaluating Language Models for Harmful Manipulation

arXiv: https://arxiv.org/abs/2603.25326 · HTML: https://mlsanity.com/paper/2603.25326/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 43 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI](https://mlsanity.com/paper/2404.15058/) (2024)
- [DarkBench: Benchmarking Dark Patterns in Large Language Models](https://mlsanity.com/paper/2503.10728/) (2025)
- [BeHonest: Benchmarking Honesty in Large Language Models](https://mlsanity.com/paper/2406.13261/) (2024)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios](https://mlsanity.com/paper/2510.15501/) (2025)
- [When Large Language Models are More PersuasiveThan Incentivized Humans, and Why](https://mlsanity.com/paper/2505.09662/) (2025)
- [OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation](https://mlsanity.com/paper/2504.13707/) (2025)
- [Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework](https://mlsanity.com/paper/2507.12872/) (2025)
- [Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments](https://mlsanity.com/paper/2510.13011/) (2025)

## Built on by (most cited first, 6 of 6)

- [EUDAIMONIA: Evaluating Undesirable Dynamics in AI](https://mlsanity.com/paper/2605.30654/) (2026)
- [DECOR: Auditing LLM Deception via Information Manipulation Theory](https://mlsanity.com/paper/2605.19270/) (2026)
- [InfoOps Bench: A live information operations safety benchmark](https://mlsanity.com/paper/2607.28503/) (2026)
- [Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactions](https://mlsanity.com/paper/2608.02491/) (2026)
- [DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs](https://mlsanity.com/paper/2608.26119/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)

## Abstract (arXiv)

Interest in the concept of AI-driven harmful manipulation is growing, yet current approaches to evaluating it are limited. This paper introduces a framework for evaluating harmful AI manipulation via context-specific human-AI interaction studies. We illustrate the utility of this framework by assessing an AI model with 10,101 participants spanning interactions in three AI use domains (public policy, finance, and health) and three locales (US, UK, and India). Overall, we find that that the tested model can produce manipulative behaviours when prompted to do so and, in experimental settings, is able to induce belief and behaviour changes in study participants. We further find that context matters: AI manipulation differs between domains, suggesting that it needs to be evaluated in the high-stakes context(s) in which an AI system is likely to be used. We also identify significant differences across our tested geographies, suggesting that AI manipulation results from one geographic region may not generalise to others. Finally, we find that the frequency of manipulative behaviours (propensity) of an AI model is not consistently predictive of the likelihood of manipulative success (efficacy), underscoring the importance of studying these dimensions separately. To facilitate adoption of our evaluation framework, we detail our testing protocols and make relevant materials publicly available. We conclude by discussing open challenges in evaluating harmful manipulation by AI models.