# How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialogues

arXiv: https://arxiv.org/abs/2504.21800 · HTML: https://mlsanity.com/paper/2504.21800/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 12 of 12)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling](https://mlsanity.com/paper/2405.16433/) (2024)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory](https://mlsanity.com/paper/2407.03103/) (2024)
- [Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions](https://mlsanity.com/paper/2408.15787/) (2024)
- [PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling](https://mlsanity.com/paper/2412.13660/) (2024)
- [Thousand Voices of Trauma: A Large-Scale Synthetic Dataset for Modeling Prolonged Exposure Therapy Conversations](https://mlsanity.com/paper/2504.13955/) (2025)
- [Privacy Sensitive Speech Analysis Using Federated Learning to Assess Depression](https://mlsanity.com/paper/2205.00111/) (2022)
- [Enhancing Psychotherapy Counseling: A Data Augmentation Pipeline Leveraging Large Language Models for Counseling Conversations](https://mlsanity.com/paper/2406.08718/) (2024)
- [WundtGPT: Shaping Large Language Models To Be An Empathetic, Proactive Psychologist](https://mlsanity.com/paper/2406.15474/) (2024)

## Built on by (most cited first, 8 of 8)

- [The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support](https://mlsanity.com/paper/2505.15065/) (2025)
- [Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elements](https://mlsanity.com/paper/2506.09707/) (2025)
- [Responsible Evaluation of AI for Mental Health](https://mlsanity.com/paper/2602.00065/) (2026)
- [AI Safety Training Can be Clinically Harmful](https://mlsanity.com/paper/2604.23445/) (2026)
- [SafeTalkCoach: Diversity-Driven Multi-Agent Simulation for Parent-Teen Health Conversations](https://mlsanity.com/paper/2602.00017/) (2026)
- [CBT-Audio: Evaluating Audio Language Models for Patient-Side Distress Intensity Estimation in CBT Session Recordings](https://mlsanity.com/paper/2605.17370/) (2026)
- [EmoTrack: Clinical-Semantic Modeling for Text-Based Depression Severity Estimation](https://mlsanity.com/paper/2605.22286/) (2026)
- [TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns](https://mlsanity.com/paper/2609.38371/) (2026)

## Abstract (arXiv)

Synthetic data adoption in healthcare is driven by privacy concerns, data access limitations, and high annotation costs. We explore synthetic Prolonged Exposure (PE) therapy conversations for PTSD as a scalable alternative for training clinical models. We systematically compare real and synthetic dialogues using linguistic, structural, and protocol-specific metrics like turn-taking and treatment fidelity. We introduce and evaluate PE-specific metrics, offering a novel framework for assessing clinical fidelity beyond surface fluency. Our findings show that while synthetic data successfully mitigates data scarcity and protects privacy, capturing the most subtle therapeutic dynamics remains a complex challenge. Synthetic dialogues successfully replicate key linguistic features of real conversations, for instance, achieving a similar Readability Score (89.2 vs. 88.1), while showing differences in some key fidelity markers like distress monitoring. This comparison highlights the need for fidelity-aware metrics that go beyond surface fluency to identify clinically significant nuances. Our model-agnostic framework is a critical tool for developers and clinicians to benchmark generative model fidelity before deployment in sensitive applications. Our findings help clarify where synthetic data can effectively complement real-world datasets, while also identifying areas for future refinement.