# Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

arXiv: https://arxiv.org/abs/2604.00014 · HTML: https://mlsanity.com/paper/2604.00014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://mlsanity.com/paper/2403.17336/) (2024)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing](https://mlsanity.com/paper/2511.03980/) (2025)
- [When Wording Steers the Evaluation: Framing Bias in LLM judges](https://mlsanity.com/paper/2601.13537/) (2026)
- [Assessing the Quality of Mental Health Support in LLM Responses through Multi-Attribute Human Evaluation](https://mlsanity.com/paper/2601.18630/) (2026)
- [Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures](https://mlsanity.com/paper/2509.08839/) (2025)

## Abstract (arXiv)

Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.