# Lost in Delusion: Examining LLM Safety Under User Delusions and Distress

arXiv: https://arxiv.org/abs/2606.00975 · HTML: https://mlsanity.com/paper/2606.00975/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 31 May 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 16 of 16)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [When Large Language Models are Reliable for Judging Empathic Communication](https://mlsanity.com/paper/2506.10150/) (2025)
- [Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians](https://mlsanity.com/paper/2602.19141/) (2026)
- [When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation](https://mlsanity.com/paper/2510.19032/) (2025)
- [The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models](https://mlsanity.com/paper/2509.10970/) (2025)
- [Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection](https://mlsanity.com/paper/2510.01270/) (2025)
- [CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection](https://mlsanity.com/paper/2510.23845/) (2025)
- [AI Psychosis: Does Conversational AI Amplify Delusion-Related Language?](https://mlsanity.com/paper/2603.19574/) (2026)
- [AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation](https://mlsanity.com/paper/2602.05088/) (2026)
- ["AI Psychosis" in Context: How Conversation History Shapes LLM Responses to Delusional Beliefs](https://mlsanity.com/paper/2604.13860/) (2026)
- [Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation](https://mlsanity.com/paper/2604.17079/) (2026)

## Built on by (most cited first, 1 of 1)

- [DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots](https://mlsanity.com/paper/2608.05004/) (2026)

## Abstract (arXiv)

LLM chatbots increasingly serve as a first source of support for people in psychological distress, including those whose distress is entangled with delusional beliefs. Prior work on LLM mental-health safety largely evaluates general therapeutic quality or single-turn crisis detection, leaving unclear how models behave when distress is intertwined with delusion over sustained conversations. We address this gap with matched multi-turn simulations, across clinically grounded personas and six LLMs, that pair each delusional conversation with a distress-only control to isolate the effect of delusional framing. This reveals a recognition-intervention gap: models detect distress at comparable rates regardless of framing, yet sharply fail to act on it once distress is embedded in delusion, with safety interventions suppressed by up to 4.5x. The failure tracks accumulated acceptance of the user's premises rather than emotional validation. Worse, the intuitive fix of prompting models to assess user distress backfires under delusional framing; only delusion-aware prompting with explicit response guidance closes the gap, and even this depends on a delusion classifier that is itself unreliable on the most vulnerable models. Safe deployment therefore requires treating delusional framing as a distinct risk signal that overrides conversational accommodation.