# Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Use

arXiv: https://arxiv.org/abs/2410.19155 · HTML: https://mlsanity.com/paper/2410.19155/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 70 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction](https://mlsanity.com/paper/2305.02466/) (2023)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [Empath: Understanding Topic Signals in Large-Scale Text](https://mlsanity.com/paper/1602.06979/) (2016)
- [Multimodal LLMs for health grounded in individual-specific data](https://mlsanity.com/paper/2307.09018/) (2023)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications](https://mlsanity.com/paper/2409.07314/) (2024)
- ["Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groups](https://mlsanity.com/paper/2102.03870/) (2021)
- [Analyzing Diversity in Healthcare LLM Research: A Scientometric Perspective](https://mlsanity.com/paper/2406.13152/) (2024)

## Built on by (most cited first, 17 of 17)

- [From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents](https://mlsanity.com/paper/2412.07951/) (2024)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [Linguistic Comparison of AI- and Human-Written Responses to Online Mental Health Queries](https://mlsanity.com/paper/2504.09271/) (2025)
- [Interpersonal Theory of Suicide as a Lens to Examine Suicidal Ideation in Online Spaces](https://mlsanity.com/paper/2504.13277/) (2025)
- [MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platform](https://mlsanity.com/paper/2506.00308/) (2025)
- [Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming](https://mlsanity.com/paper/2602.19948/) (2026)
- [Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies](https://mlsanity.com/paper/2503.07306/) (2025)
- [A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Models](https://mlsanity.com/paper/2504.02793/) (2025)
- [Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation](https://mlsanity.com/paper/2510.16829/) (2025)
- [Responsible Evaluation of AI for Mental Health](https://mlsanity.com/paper/2602.00065/) (2026)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [Towards Experience-Centered AI: A Framework for Integrating Lived Experience in Design and Development](https://mlsanity.com/paper/2508.06849/) (2025)
- [Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study](https://mlsanity.com/paper/2604.07558/) (2026)
- [Navigating the Rabbit Hole: Emergent Biases in LLM-Generated Attack Narratives Targeting Mental Health Groups](https://mlsanity.com/paper/2504.06160/) (2025)
- [HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs](https://mlsanity.com/paper/2507.21815/) (2025)
- [CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives](https://mlsanity.com/paper/2609.19585/) (2026)
- [Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation](https://mlsanity.com/paper/2609.21075/) (2026)

## Abstract (arXiv)

Adverse Drug Reactions (ADRs) from psychiatric medications are the leading cause of hospitalizations among mental health patients. With healthcare systems and online communities facing limitations in resolving ADR-related issues, Large Language Models (LLMs) have the potential to fill this gap. Despite the increasing capabilities of LLMs, past research has not explored their capabilities in detecting ADRs related to psychiatric medications or in providing effective harm reduction strategies. To address this, we introduce the Psych-ADR benchmark and the Adverse Drug Reaction Response Assessment (ADRA) framework to systematically evaluate LLM performance in detecting ADR expressions and delivering expert-aligned mitigation strategies. Our analyses show that LLMs struggle with understanding the nuances of ADRs and differentiating between types of ADRs. While LLMs align with experts in terms of expressed emotions and tone of the text, their responses are more complex, harder to read, and only 70.86% aligned with expert strategies. Furthermore, they provide less actionable advice by a margin of 12.32% on average. Our work provides a comprehensive benchmark and evaluation framework for assessing LLMs in strategy-driven tasks within high-risk domains.