# SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2208.13077 · HTML: https://mlsanity.com/paper/2208.13077/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 73 references that are in the map.
- First posted on arXiv 27 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/1811.00855/) (2018)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Evaluation of Session-based Recommendation Algorithms](https://mlsanity.com/paper/1803.09587/) (2018)
- [Neural Topic Modeling with Bidirectional Adversarial Training](https://mlsanity.com/paper/2004.12331/) (2020)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [Deep Annotation of Therapeutic Working Alliance in Psychotherapy](https://mlsanity.com/paper/2204.05522/) (2022)
- [A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry](https://mlsanity.com/paper/1906.11286/) (2019)
- [Neural Topic Modeling of Psychotherapy Sessions](https://mlsanity.com/paper/2204.10189/) (2022)
- [Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL](https://mlsanity.com/paper/2005.04544/) (2020)
- [Predicting human decision making in psychological tasks with recurrent neural networks](https://mlsanity.com/paper/2010.11413/) (2020)
- [Split Q Learning: Reinforcement Learning with Two-Stream Rewards](https://mlsanity.com/paper/1906.12350/) (2019)
- [Speaker Diarization as a Fully Online Learning Problem in MiniVox](https://mlsanity.com/paper/2006.04376/) (2020)
- [Knowledge Management System with NLP-Assisted Annotations: A Brief Survey and Outlook](https://mlsanity.com/paper/2206.07304/) (2022)
- [Online Semi-Supervised Learning in Contextual Bandits with Episodic Reward](https://mlsanity.com/paper/2009.08457/) (2020)

## Built on by (most cited first, 8 of 8)

- [Towards Healthy AI: Large Language Models Need Therapists Too](https://mlsanity.com/paper/2304.00416/) (2023)
- [A Survey on Compositional Generalization in Applications](https://mlsanity.com/paper/2302.01067/) (2023)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Working Alliance Transformer for Psychotherapy Dialogue Classification](https://mlsanity.com/paper/2210.15603/) (2022)
- [Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics](https://mlsanity.com/paper/2303.09601/) (2023)
- [Computational Inference in Cognitive Science: Operational, Societal and Ethical Considerations](https://mlsanity.com/paper/2210.13526/) (2022)
- [Utterance Classification with Logical Neural Network: Explainable AI for Mental Disorder Diagnosis](https://mlsanity.com/paper/2306.03902/) (2023)
- [TherapyView: Visualizing Therapy Sessions with Temporal Topic Modeling and AI-Generated Arts](https://mlsanity.com/paper/2302.10845/) (2023)

## Abstract (arXiv)

We propose a recommendation system that suggests treatment strategies to a therapist during the psychotherapy session in real-time. Our system uses a turn-level rating mechanism that predicts the therapeutic outcome by computing a similarity score between the deep embedding of a scoring inventory, and the current sentence that the patient is speaking. The system automatically transcribes a continuous audio stream and separates it into turns of the patient and of the therapist and perform real-time inference of their therapeutic working alliance. The dialogue pairs along with their computed working alliance as ratings are then fed into a deep reinforcement learning recommendation system where the sessions are treated as users and the topics are treated as items. Other than evaluating the empirical advantages of the core components on an existing dataset of psychotherapy sessions, we demonstrate the effectiveness of this system in a web app.