# ChatbotManip: A Dataset to Facilitate Evaluation and Oversight of Manipulative Chatbot Behaviour

arXiv: https://arxiv.org/abs/2506.12090 · HTML: https://mlsanity.com/paper/2506.12090/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 35 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 6 of 6)

- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI](https://mlsanity.com/paper/2404.15058/) (2024)
- [MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations](https://mlsanity.com/paper/2405.16584/) (2024)
- [Detecting Conversational Mental Manipulation with Intent-Aware Prompting](https://mlsanity.com/paper/2412.08414/) (2024)
- [Resistance Against Manipulative AI: key factors and possible actions](https://mlsanity.com/paper/2404.14230/) (2024)
- [Enhanced Detection of Conversational Mental Manipulation Through Advanced Prompting Techniques](https://mlsanity.com/paper/2408.07676/) (2024)

## Built on by (most cited first, 4 of 4)

- [The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues](https://mlsanity.com/paper/2603.20907/) (2026)
- [LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight](https://mlsanity.com/paper/2605.08321/) (2026)
- [CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model](https://mlsanity.com/paper/2606.06099/) (2026)
- [AI Watchdog: Agent Interfaces for Detecting and Defending Against Manipulative Dark Patterns in AI Conversations](https://mlsanity.com/paper/2608.21841/) (2026)

## Abstract (arXiv)

This paper introduces ChatbotManip, a novel dataset for studying manipulation in Chatbots. It contains simulated generated conversations between a chatbot and a (simulated) user, where the chatbot is explicitly asked to showcase manipulation tactics, persuade the user towards some goal, or simply be helpful. We consider a diverse set of chatbot manipulation contexts, from consumer and personal advice to citizen advice and controversial proposition argumentation. Each conversation is annotated by human annotators for both general manipulation and specific manipulation tactics. Our research reveals three key findings. First, Large Language Models (LLMs) can be manipulative when explicitly instructed, with annotators identifying manipulation in approximately 84% of such conversations. Second, even when only instructed to be ``persuasive'' without explicit manipulation prompts, LLMs frequently default to controversial manipulative strategies, particularly gaslighting and fear enhancement. Third, small fine-tuned open source models, such as BERT+BiLSTM have a performance comparable to zero-shot classification with larger models like Gemini 2.5 pro in detecting manipulation, but are not yet reliable for real-world oversight. Our work provides important insights for AI safety research and highlights the need of addressing manipulation risks as LLMs are increasingly deployed in consumer-facing applications.