# TherapyProbe: Generating Design Knowledge for Relational Safety in Mental Health Chatbots Through Adversarial Simulation

arXiv: https://arxiv.org/abs/2602.22775 · HTML: https://mlsanity.com/paper/2602.22775/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 52 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety](https://mlsanity.com/paper/2504.09689/) (2025)
- [CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering](https://mlsanity.com/paper/2506.08584/) (2025)
- [TRACE: Transparent Web Reliability Assessment with Contextual Explanations](https://mlsanity.com/paper/2506.12072/) (2025)
- [Advancing Responsible Innovation in Agentic AI: A study of Ethical Frameworks for Household Automation](https://mlsanity.com/paper/2507.15901/) (2025)
- [Evaluating an LLM-Powered Chatbot for Cognitive Restructuring: Insights from Mental Health Professionals](https://mlsanity.com/paper/2501.15599/) (2025)
- [Orchestrating Attention: Bringing Harmony to the 'Chaos' of Neurodivergent Learning States](https://mlsanity.com/paper/2602.07865/) (2026)

## Abstract (arXiv)

As mental health chatbots proliferate to address the global treatment gap, a critical question emerges: How do we design for relational safety the quality of interaction patterns that unfold across conversations rather than the correctness of individual responses? Current safety evaluations assess single-turn crisis responses, missing the therapeutic dynamics that determine whether chatbots help or harm over time. We introduce TherapyProbe, a design probe methodology that generates actionable design knowledge by systematically exploring chatbot conversation trajectories through adversarial multi-agent simulation. Using open-source models, TherapyProbe surfaces relational safety failures interaction patterns like "validation spirals" where chatbots progressively reinforce hopelessness, or "empathy fatigue" where responses become mechanical over turns. Our contribution is translating these failures into a Safety Pattern Library of 23 failure archetypes with corresponding design recommendations. We contribute: (1) a replicable methodology requiring no API costs, (2) a clinically-grounded failure taxonomy, and (3) design implications for developers, clinicians, and policymakers.