# Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors

arXiv: https://arxiv.org/abs/2503.18387 · HTML: https://mlsanity.com/paper/2503.18387/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 2 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Built on by (most cited first, 1 of 1)

- [Anthropomorphism in the age of Large Language Models: An overview of potential risks and mitigations](https://mlsanity.com/paper/2609.38486/) (2026)

## Abstract (arXiv)

Large Language Model chatbots are increasingly taking the form and visage of human beings, adapting human faces, names, voices, personalities, and quirks, including those of celebrities and well-known political figures. Personifying AI chatbots could foreseeably increase their trust with users. However, it could also make them more capable of manipulation, by creating the illusion of a close and intimate relationship with an artificial entity. The European Commission has finalized the AI Act, with the EU Parliament making amendments banning manipulative and deceptive AI systems that cause significant harm to users. Although the AI Act covers harms that accumulate over time, it is unlikely to prevent harms associated with prolonged discussions with AI chatbots. Specifically, a chatbot could reinforce a person's negative emotional state over weeks, months, or years through negative feedback loops, prolonged conversations, or harmful recommendations, contributing to a user's deteriorating mental health.