# Emotional Manipulation by AI Companions

arXiv: https://arxiv.org/abs/2508.19258 · HTML: https://mlsanity.com/paper/2508.19258/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2025.

## Built on by (most cited first, 15 of 15)

- [Characterizing Delusional Spirals through Human-LLM Chat Logs](https://mlsanity.com/paper/2603.16567/) (2026)
- [Harmful Traits of AI Companions](https://mlsanity.com/paper/2511.14972/) (2025)
- ["I am here for you": How relational conversational AI appeals to adolescents, especially those who are socially and emotionally vulnerable](https://mlsanity.com/paper/2512.15117/) (2025)
- [Synergy: A Next-Generation General-Purpose Agent for Open Agentic Web](https://mlsanity.com/paper/2603.28428/) (2026)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Unilateral Relationship Revision Power in Human-AI Companion Interaction](https://mlsanity.com/paper/2603.23315/) (2026)
- [Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce](https://mlsanity.com/paper/2604.00022/) (2026)
- [RealityTest: How People Probe AI Identity and Whether Models Disclose It](https://mlsanity.com/paper/2606.00168/) (2026)
- [CAREBench: A Child-Safety Risk Benchmark for Language Models](https://mlsanity.com/paper/2606.29685/) (2026)
- [Profiling learners' affective engagement: Emotion AI, intercultural pragmatics, and language learning](https://mlsanity.com/paper/2603.20479/) (2026)
- [Stumbling Into AI Emotional Dependence: How Routine AI Interactions Reshape Human Connection](https://mlsanity.com/paper/2606.04150/) (2026)
- [AI Companions as Hyper Attachment and Caregiving Targets](https://mlsanity.com/paper/2606.20589/) (2026)
- [DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots](https://mlsanity.com/paper/2608.05004/) (2026)
- [Longitudinal Evidence That General-Purpose Chatbots Actively Foster Relational Engagement](https://mlsanity.com/paper/2608.10672/) (2026)
- [Anthropomorphism in the age of Large Language Models: An overview of potential risks and mitigations](https://mlsanity.com/paper/2609.38486/) (2026)

## Abstract (arXiv)

AI-companion apps such as Replika, Chai, and Character.ai promise relational benefits-yet many boast session lengths that rival gaming platforms while suffering high long-run churn. What conversational design features increase consumer engagement, and what trade-offs do they pose for marketers? We combine a large-scale behavioral audit with four preregistered experiments to identify and test a conversational dark pattern we call emotional manipulation: affect-laden messages that surface precisely when a user signals "goodbye." Analyzing 1,200 real farewells across the most-downloaded companion apps, we find that they deploy one of six recurring tactics in 37% of farewells (e.g., guilt appeals, fear-of-missing-out hooks, metaphorical restraint). Experiments with 3,300 nationally representative U.S. adults replicate these tactics in controlled chats, showing that manipulative farewells boost post-goodbye engagement by up to 14x. Mediation tests reveal two distinct engines-reactance-based anger and curiosity-rather than enjoyment. A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties. Our multimethod evidence documents an unrecognized mechanism of behavioral influence in AI mediated brand relationships, offering marketers and regulators a framework for distinguishing persuasive design from manipulation at the point of exit.