mlsanityData to 14 Oct 2026

From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents

arXiv 2412.07951 ↗First posted 10 Dec 2024cs.HC cs.AI cs.CYcited by 25

Not on the map. Why?

Abstract (arXiv)

Recent gains in popularity of AI conversational agents have led to their increased use for improving productivity and supporting well-being. While previous research has aimed to understand the risks associated with interactions with AI conversational agents, these studies often fall short in capturing the lived experiences of individuals. Additionally, psychological risks have often been presented as a sub-category within broader AI-related risks in past taxonomy works, leading to under-representation of the impact of psychological risks of AI use. To address these challenges, our work presents a novel risk taxonomy focusing on psychological risks of using AI gathered through the lived experiences of individuals. We employed a mixed-method approach, involving a comprehensive survey with 283 people with lived mental health experience and workshops involving experts with lived experience to develop a psychological risk taxonomy. Our taxonomy features 19 AI behaviors, 21 negative psychological impacts, and 15 contexts related to individuals. Additionally, we propose a novel multi-path vignette-based framework for understanding the complex interplay between AI behaviors, psychological impacts, and individual user contexts. Finally, based on the feedback obtained from the workshop sessions, we present design recommendations for developing safer and more robust AI agents. Our work offers an in-depth understanding of the psychological risks associated with AI conversational agents and provides actionable recommendations for policymakers, researchers, and developers.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 41 of its 136 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

41 of 136 references.

References · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  12. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
29 more
  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  6. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  8. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  9. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  10. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  11. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  12. 2024A Roadmap to Pluralistic Alignmentcited by 184
  13. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  14. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  15. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
  16. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  17. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  18. 2024The Ethics of Advanced AI Assistantscited by 107
  19. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  20. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 62
  21. 2023"I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generationcited by 56
  22. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  23. 2023Characterizing Manipulation from AI Systemscited by 43
  24. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 40
  25. 2023The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answeringcited by 26
  26. 2024Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Usecited by 17
  27. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15
  28. 2022A Machine with Short-Term, Episodic, and Semantic Memory Systemscited by 9
  29. 2023Regulating AI: Applying insights from behavioural economics and psychology to the application of article 5 of the EU AI Actcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 15.

25 papers.

Sort:

By area:

  1. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  2. 2025Measuring Agents in Productioncited by 37
  3. 2026Characterizing Delusional Spirals through Human-LLM Chat Logscited by 23
  4. 2025Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systemscited by 11
  5. 2025Longitudinal Study on Social and Emotional Use of AI Conversational Agentcited by 10
  6. 2026RubRIX: Rubric-Driven Risk Mitigation in Caregiver-AI Interactionscited by 10
  7. 2025Mental Health Impacts of AI Companions: Triangulating Social Media Quasi-Experiments, User Perspectives, and Relational Theorycited by 9
  8. 2026PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Supportcited by 7
  9. 2025A Risk Ontology for Evaluating AI-Powered Psychotherapy Virtual Agentscited by 5
  10. 2026AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluationcited by 4
  11. 2025When AI Gives Advice: Evaluating AI and Human Responses to Online Advice-Seeking for Well-Beingcited by 3
  12. 2026Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teamingcited by 3
13 more
  1. 2026MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Modelscited by 3
  2. 2025Seeking Late Night Life Lines: Experiences of Conversational AI Use in Mental Health Crisiscited by 2
  3. 2026Brainrot: Deskilling and Addiction are Overlooked AI Riskscited by 2
  4. 2026LLUMI: Improving LLM Writing Assistance for Mental Health Support with Online Community Feedbackcited by 2
  5. 2025AI Chaperones Are (Really) All You Need to Prevent Parasocial Relationships with Chatbotscited by 1
  6. 2025Lived Experience in Dialogue: Co-designing Personalization in Large Language Models to Support Youth Mental Well-beingcited by 0
  7. 2026Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contextscited by 0
  8. 2026EthicMind: A Risk-Aware Framework for Ethical-Emotional Alignment in Multi-Turn Dialoguecited by 0
  9. 2026Understanding How International Students in the U.S. Are Using Conversational AI to Support Cross-Cultural Adaptationcited by 0
  10. 2026Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Promptscited by 0
  11. 2026Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Beingcited by 0
  12. 2026PeerPen: AI-Assisted Writing for Online Mental Health Peer Supportcited by 0
  13. 2026Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generationcited by 0