# AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression

arXiv: https://arxiv.org/abs/2504.18932 · HTML: https://mlsanity.com/paper/2504.18932/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 133 references that are in the map.
- First posted on arXiv 26 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support](https://mlsanity.com/paper/2307.15810/) (2023)
- [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023)
- [Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation](https://mlsanity.com/paper/2310.12953/) (2023)
- [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024)
- [Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuring](https://mlsanity.com/paper/2310.15461/) (2023)
- [MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling](https://mlsanity.com/paper/2310.05231/) (2023)
- [How Different Groups Prioritize Ethical Values for Responsible AI](https://mlsanity.com/paper/2205.07722/) (2022)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory](https://mlsanity.com/paper/2205.05057/) (2022)
- [Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks](https://mlsanity.com/paper/2310.07879/) (2023)
- [Designing for Human-Agent Alignment: Understanding what humans want from their agents](https://mlsanity.com/paper/2404.04289/) (2024)
- [Chatbots for Mental Health Support: Exploring the Impact of Emohaa on Reducing Mental Distress in China](https://mlsanity.com/paper/2209.10183/) (2022)
- [Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders](https://mlsanity.com/paper/2303.06794/) (2023)
- [Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits](https://mlsanity.com/paper/2403.00145/) (2024)
- [The Future of HCI-Policy Collaboration](https://mlsanity.com/paper/2409.19738/) (2024)
- [Towards Understanding Emotions for Engaged Mental Health Conversations](https://mlsanity.com/paper/2406.11135/) (2024)
- [Reconciling Governmental Use of Online Targeting With Democracy](https://mlsanity.com/paper/2306.01479/) (2023)

## Built on by (most cited first, 13 of 13)

- [Mapping Caregiver Needs to AI Chatbot Design: Strengths and Gaps in Mental Health Support for Alzheimer's and Dementia Caregivers](https://mlsanity.com/paper/2506.15047/) (2025)
- [Understanding Teen Overreliance on AI Companion Chatbots Through Self-Reported Reddit Narratives](https://mlsanity.com/paper/2507.15783/) (2025)
- [RubRIX: Rubric-Driven Risk Mitigation in Caregiver-AI Interactions](https://mlsanity.com/paper/2601.13235/) (2026)
- [Mental Health Impacts of AI Companions: Triangulating Social Media Quasi-Experiments, User Perspectives, and Relational Theory](https://mlsanity.com/paper/2509.22505/) (2025)
- [Balancing Caregiving and Self-Care: Exploring Mental Health Needs of Alzheimer's and Dementia Caregivers](https://mlsanity.com/paper/2506.14196/) (2025)
- [PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Support](https://mlsanity.com/paper/2601.12754/) (2026)
- [AI Psychosis: Does Conversational AI Amplify Delusion-Related Language?](https://mlsanity.com/paper/2603.19574/) (2026)
- [Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation](https://mlsanity.com/paper/2509.14760/) (2025)
- [Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiency](https://mlsanity.com/paper/2511.07277/) (2025)
- [Designing KRIYA: An AI Companion for Wellbeing Self-Reflection](https://mlsanity.com/paper/2601.14589/) (2026)
- [A Checklist for Trustworthy, Safe, and User-Friendly Mental Health Chatbots](https://mlsanity.com/paper/2601.15412/) (2026)
- ["I Said Things I Needed to Hear Myself": Peer Support as an Emotional, Organisational, and Sociotechnical Practice in Singapore](https://mlsanity.com/paper/2506.09362/) (2025)
- [AI Content Moderation in Therapy Conversations](https://mlsanity.com/paper/2605.25454/) (2026)

## Abstract (arXiv)

Recent advancements in LLMs enable chatbots to interact with individuals on a range of queries, including sensitive mental health contexts. Despite uncertainties about their effectiveness and reliability, the development of LLMs in these areas is growing, potentially leading to harms. To better identify and mitigate these harms, it is critical to understand how the values of people with lived experiences relate to the harms. In this study, we developed a technology probe, a GPT-4o based chatbot called Zenny, enabling participants to engage with depression self-management scenarios informed by previous research. We used Zenny to interview 17 individuals with lived experiences of depression. Our thematic analysis revealed key values: informational support, emotional support, personalization, privacy, and crisis management. This work explores the relationship between lived experience values, potential harms, and design recommendations for mental health AI chatbots, aiming to enhance self-management support while minimizing risks.