# The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues

arXiv: https://arxiv.org/abs/2603.20907 · HTML: https://mlsanity.com/paper/2603.20907/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships](https://mlsanity.com/paper/2410.20130/) (2024)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Language](https://mlsanity.com/paper/2406.17753/) (2024)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations](https://mlsanity.com/paper/2405.16584/) (2024)
- [Human Decision-making is Susceptible to AI-driven Manipulation](https://mlsanity.com/paper/2502.07663/) (2025)
- [The Call for Socially Aware Language Technologies](https://mlsanity.com/paper/2405.02411/) (2024)
- [An Unsupervised Domain-Independent Framework for Automated Detection of Persuasion Tactics in Text](https://mlsanity.com/paper/1912.06745/) (2019)
- [A Meta-Analysis of the Persuasive Power of Large Language Models](https://mlsanity.com/paper/2512.01431/) (2025)
- [Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication](https://mlsanity.com/paper/2505.21451/) (2025)
- [ChatbotManip: A Dataset to Facilitate Evaluation and Oversight of Manipulative Chatbot Behaviour](https://mlsanity.com/paper/2506.12090/) (2025)
- [How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversations](https://mlsanity.com/paper/2508.05625/) (2025)
- [ManiTweet: A New Benchmark for Identifying Manipulation of News on Social Media](https://mlsanity.com/paper/2305.14225/) (2023)
- [SELF-PERCEPT: Introspection Improves Large Language Models' Detection of Multi-Person Mental Manipulation in Conversations](https://mlsanity.com/paper/2505.20679/) (2025)
- [CLAIM: An Intent-Driven Multi-Agent Framework for Analyzing Manipulation in Courtroom Dialogues](https://mlsanity.com/paper/2506.04131/) (2025)
- [DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views](https://mlsanity.com/paper/2603.10018/) (2026)

## Built on by (most cited first, 2 of 2)

- [LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight](https://mlsanity.com/paper/2605.08321/) (2026)
- [Annie, Are You Okay? How Style- and Context-Based Personalization Shape AI-Assisted Decision-Making](https://mlsanity.com/paper/2609.24644/) (2026)

## Abstract (arXiv)

As users increasingly turn to LLMs for practical and personal advice, they become vulnerable to subtle steering toward hidden incentives misaligned with their own interests. While existing NLP research has benchmarked manipulation detection, these efforts often rely on simulated debates and remain fundamentally decoupled from actual human belief shifts in real-world scenarios. We introduce PUPPET, a theoretical taxonomy and resource that bridges this gap by focusing on the moral direction of hidden incentives in everyday, advice-giving contexts. We provide an evaluation dataset of N=1,035 human-LLM interactions, where we measure users' belief shifts. Our analysis reveals a critical disconnect in current safety paradigms: while models can be trained to detect manipulative strategies, they do not correlate with the magnitude of resulting belief change. As such, we define the task of human belief shift prediction and show that while state-of-the-art LLMs achieve moderate correlation (r=0.3-0.5), they exhibit systematic directional biases, with certain models over or under-predicting the magnitude of human belief change. This work establishes a theoretically grounded and behaviorally validated foundation for AI social safety efforts by studying incentive-driven manipulation in LLMs during everyday, practical user queries.