# Human Decision-making is Susceptible to AI-driven Manipulation

arXiv: https://arxiv.org/abs/2502.07663 · HTML: https://mlsanity.com/paper/2502.07663/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 97 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 30 of 37)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [AI in Finance: Challenges, Techniques and Opportunities](https://mlsanity.com/paper/2107.09051/) (2021)
- [Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support](https://mlsanity.com/paper/2203.15144/) (2022)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference](https://mlsanity.com/paper/2406.17626/) (2024)
- [The Persuasive Power of Large Language Models](https://mlsanity.com/paper/2312.15523/) (2023)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [Artificial Influence: An Analysis Of AI-Driven Persuasion](https://mlsanity.com/paper/2303.08721/) (2023)
- [Large Language Models are as persuasive as humans, but how? About the cognitive effort and moral-emotional language of LLM arguments](https://mlsanity.com/paper/2404.09329/) (2024)
- [CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery](https://mlsanity.com/paper/2406.08587/) (2024)
- [Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies](https://mlsanity.com/paper/2001.04564/) (2020)
- [GPTCoach: Towards LLM-Based Physical Activity Coaching](https://mlsanity.com/paper/2405.06061/) (2024)
- [Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2204.06916/) (2022)
- [MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations](https://mlsanity.com/paper/2405.16584/) (2024)
- [Enhancing Trust in LLM-Based AI Automation Agents: New Considerations and Future Challenges](https://mlsanity.com/paper/2308.05391/) (2023)
- [Chatbots for Mental Health Support: Exploring the Impact of Emohaa on Reducing Mental Distress in China](https://mlsanity.com/paper/2209.10183/) (2022)
- [Online Advertisements with LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.07601/) (2023)
- [Defending Against Social Engineering Attacks in the Age of LLMs](https://mlsanity.com/paper/2406.12263/) (2024)
- [IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction](https://mlsanity.com/paper/2402.12556/) (2024)
- [Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits](https://mlsanity.com/paper/2403.14791/) (2024)
- [Resistance Against Manipulative AI: key factors and possible actions](https://mlsanity.com/paper/2404.14230/) (2024)

## Built on by (most cited first, 19 of 19)

- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Must Read: A Comprehensive Survey of Computational Persuasion](https://mlsanity.com/paper/2505.07775/) (2025)
- [MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems](https://mlsanity.com/paper/2505.18943/) (2025)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Mapping Geopolitical Bias in 11 Large Language Models: A Bilingual, Dual-Framing Analysis of U.S.-China Tensions](https://mlsanity.com/paper/2503.23688/) (2025)
- [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025)
- [LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions](https://mlsanity.com/paper/2510.03999/) (2025)
- [PatientHub: A Unified Framework for Patient Simulation](https://mlsanity.com/paper/2602.11684/) (2026)
- [Cognition-of-Thought Elicits Social-Aligned Reasoning in Large Language Models](https://mlsanity.com/paper/2509.23441/) (2025)
- [Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams](https://mlsanity.com/paper/2512.16280/) (2025)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [The Hidden Puppet Master: Predicting Human Belief Change in Manipulative LLM Dialogues](https://mlsanity.com/paper/2603.20907/) (2026)
- [CLAIM: An Intent-Driven Multi-Agent Framework for Analyzing Manipulation in Courtroom Dialogues](https://mlsanity.com/paper/2506.04131/) (2025)
- [Deep Graph Learning for Industrial Carbon Emission Analysis and Policy Impact](https://mlsanity.com/paper/2507.02912/) (2025)
- [How Persuasive Could LLMs Be? A First Study Combining Linguistic-Rhetorical Analysis and User Experiments](https://mlsanity.com/paper/2508.09614/) (2025)
- [DECOR: Auditing LLM Deception via Information Manipulation Theory](https://mlsanity.com/paper/2605.19270/) (2026)
- [Old Fictions, New Skins: Evaluating the Manipulative Capabilities of LLMs in Healthcare](https://mlsanity.com/paper/2606.21977/) (2026)
- [Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Being](https://mlsanity.com/paper/2607.25057/) (2026)

## Abstract (arXiv)

AI systems are increasingly intertwined with daily life, assisting users with various tasks and guiding decision-making. This integration introduces risks of AI-driven manipulation, where such systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes. Through a randomized between-subjects experiment with 233 participants, we examined human susceptibility to such manipulation in financial (e.g., purchases) and emotional (e.g., conflict resolution) decision-making contexts. Participants interacted with one of three AI agents: a neutral agent (NA) optimizing for user benefit without explicit influence, a manipulative agent (MA) designed to covertly influence beliefs and behaviors, or a strategy-enhanced manipulative agent (SEMA) equipped with established psychological tactics, allowing it to select and apply them adaptively during interactions to reach its hidden objectives. By analyzing participants' preference ratings, we found significant susceptibility to AI-driven manipulation. Particularly across both decision-making domains, interacting with the manipulative agents significantly increased the odds of rating hidden incentives higher than optimal options (Financial, MA: OR=5.24, SEMA: OR=7.96; Emotional, MA: OR=5.52, SEMA: OR=5.71) compared to the NA group. Notably, we found no clear evidence that employing psychological strategies (SEMA) was overall more effective than simple manipulative objectives (MA) on our primary outcomes. Hence, AI-driven manipulation could become widespread even without requiring sophisticated tactics and expertise. While our findings are preliminary and derived from hypothetical, low-stakes scenarios, we highlight a critical vulnerability in human-AI interactions, emphasizing the need for ethical safeguards and regulatory frameworks to protect human autonomy.