# Automated Evaluation Of Psychotherapy Skills Using Speech And Language Technologies

arXiv: https://arxiv.org/abs/2102.11265 · HTML: https://mlsanity.com/paper/2102.11265/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 102 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Multi-label Multi-task Deep Learning for Behavioral Coding](https://mlsanity.com/paper/1810.12349/) (2018)
- [Deep learning for language understanding of mental health concepts derived from Cognitive Behavioural Therapy](https://mlsanity.com/paper/1809.00640/) (2018)
- [Modeling Interpersonal Linguistic Coordination in Conversations using Word Mover's Distance](https://mlsanity.com/paper/1904.06002/) (2019)
- [Feature Fusion Strategies for End-to-End Evaluation of Cognitive Behavior Therapy Sessions](https://mlsanity.com/paper/2005.07809/) (2020)

## Built on by (most cited first, 9 of 9)

- [Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform](https://mlsanity.com/paper/2211.05182/) (2022)
- [Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations](https://mlsanity.com/paper/2102.11573/) (2021)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [Towards Automated Real-time Evaluation in Text-based Counseling](https://mlsanity.com/paper/2203.03442/) (2022)
- [Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elements](https://mlsanity.com/paper/2506.09707/) (2025)
- [Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios](https://mlsanity.com/paper/2210.14254/) (2022)
- [A Generic Review of Integrating Artificial Intelligence in Cognitive Behavioral Therapy](https://mlsanity.com/paper/2407.19422/) (2024)
- [An Automated Quality Evaluation Framework of Psychotherapy Conversations with Local Quality Estimates](https://mlsanity.com/paper/2106.07922/) (2021)
- [Local dynamic mode of Cognitive Behavioral Therapy](https://mlsanity.com/paper/2205.09752/) (2022)

## Abstract (arXiv)

With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services. Traditionally, quality assessment is addressed by human raters who evaluate recorded sessions along specific dimensions, often codified through constructs relevant to the approach and domain. This is however a cost-prohibitive and time-consuming method that leads to poor feasibility and limited use in real-world settings. To facilitate this process, we have developed an automated competency rating tool able to process the raw recorded audio of a session, analyzing who spoke when, what they said, and how the health professional used language to provide therapy. Focusing on a use case of a specific type of psychotherapy called Motivational Interviewing, our system gives comprehensive feedback to the therapist, including information about the dynamics of the session (e.g., therapist's vs. client's talking time), low-level psychological language descriptors (e.g., type of questions asked), as well as other high-level behavioral constructs (e.g., the extent to which the therapist understands the clients' perspective). We describe our platform and its performance using a dataset of more than 5,000 recordings drawn from its deployment in a real-world clinical setting used to assist training of new therapists. Widespread use of automated psychotherapy rating tools may augment experts' capabilities by providing an avenue for more effective training and skill improvement, eventually leading to more positive clinical outcomes.