# Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

arXiv: https://arxiv.org/abs/2303.09601 · HTML: https://mlsanity.com/paper/2303.09601/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/1811.00855/) (2018)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Evaluation of Session-based Recommendation Algorithms](https://mlsanity.com/paper/1803.09587/) (2018)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [Deep Annotation of Therapeutic Working Alliance in Psychotherapy](https://mlsanity.com/paper/2204.05522/) (2022)
- [A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry](https://mlsanity.com/paper/1906.11286/) (2019)
- [Neural Topic Modeling of Psychotherapy Sessions](https://mlsanity.com/paper/2204.10189/) (2022)
- [Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL](https://mlsanity.com/paper/2005.04544/) (2020)
- [Predicting human decision making in psychological tasks with recurrent neural networks](https://mlsanity.com/paper/2010.11413/) (2020)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Working Alliance Transformer for Psychotherapy Dialogue Classification](https://mlsanity.com/paper/2210.15603/) (2022)
- [SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning](https://mlsanity.com/paper/2208.13077/) (2022)
- [Speaker Diarization as a Fully Online Learning Problem in MiniVox](https://mlsanity.com/paper/2006.04376/) (2020)
- [Knowledge Management System with NLP-Assisted Annotations: A Brief Survey and Outlook](https://mlsanity.com/paper/2206.07304/) (2022)
- [Computational Inference in Cognitive Science: Operational, Societal and Ethical Considerations](https://mlsanity.com/paper/2210.13526/) (2022)
- [TherapyView: Visualizing Therapy Sessions with Temporal Topic Modeling and AI-Generated Arts](https://mlsanity.com/paper/2302.10845/) (2023)
- [A Reinforcement Learning Framework for Online Speaker Diarization](https://mlsanity.com/paper/2302.10924/) (2023)

## Built on by (most cited first, 6 of 6)

- [Towards Healthy AI: Large Language Models Need Therapists Too](https://mlsanity.com/paper/2304.00416/) (2023)
- [Survey: Multi-Armed Bandits Meet Large Language Models](https://mlsanity.com/paper/2505.13355/) (2025)
- [Utterance Classification with Logical Neural Network: Explainable AI for Mental Disorder Diagnosis](https://mlsanity.com/paper/2306.03902/) (2023)
- [A Scoping Review of AI-Driven Digital Interventions in Mental Health Care: Mapping Applications Across Screening, Support, Monitoring, Prevention, and Clinical Education](https://mlsanity.com/paper/2603.16204/) (2026)
- [The Pros and Cons of Using Machine Learning and Interpretable Machine Learning Methods in psychiatry detection applications, specifically depression disorder: A Brief Review](https://mlsanity.com/paper/2311.06633/) (2023)
- [Word Clouds as Common Voices: LLM-Assisted Visualization of Participant-Weighted Themes in Qualitative Interviews](https://mlsanity.com/paper/2508.07517/) (2025)

## Abstract (arXiv)

We introduce a Reinforcement Learning Psychotherapy AI Companion that generates topic recommendations for therapists based on patient responses. The system uses Deep Reinforcement Learning (DRL) to generate multi-objective policies for four different psychiatric conditions: anxiety, depression, schizophrenia, and suicidal cases. We present our experimental results on the accuracy of recommended topics using three different scales of working alliance ratings: task, bond, and goal. We show that the system is able to capture the real data (historical topics discussed by the therapists) relatively well, and that the best performing models vary by disorder and rating scale. To gain interpretable insights into the learned policies, we visualize policy trajectories in a 2D principal component analysis space and transition matrices. These visualizations reveal distinct patterns in the policies trained with different reward signals and trained on different clinical diagnoses. Our system's success in generating DIsorder-Specific Multi-Objective Policies (DISMOP) and interpretable policy dynamics demonstrates the potential of DRL in providing personalized and efficient therapeutic recommendations.