# Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations

arXiv: https://arxiv.org/abs/2102.11573 · HTML: https://mlsanity.com/paper/2102.11573/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 78 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)
- [DocBERT: BERT for Document Classification](https://mlsanity.com/paper/1904.08398/) (2019)
- [Massive Exploration of Neural Machine Translation Architectures](https://mlsanity.com/paper/1703.03906/) (2017)
- [Multi-label Multi-task Deep Learning for Behavioral Coding](https://mlsanity.com/paper/1810.12349/) (2018)
- [Deep learning for language understanding of mental health concepts derived from Cognitive Behavioural Therapy](https://mlsanity.com/paper/1809.00640/) (2018)
- [Automated Evaluation Of Psychotherapy Skills Using Speech And Language Technologies](https://mlsanity.com/paper/2102.11265/) (2021)
- [Modeling Interpersonal Linguistic Coordination in Conversations using Word Mover's Distance](https://mlsanity.com/paper/1904.06002/) (2019)
- [Feature Fusion Strategies for End-to-End Evaluation of Cognitive Behavior Therapy Sessions](https://mlsanity.com/paper/2005.07809/) (2020)
- [Towards an Unsupervised Entrainment Distance in Conversational Speech using Deep Neural Networks](https://mlsanity.com/paper/1804.08782/) (2018)

## Built on by (most cited first, 13 of 13)

- [Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors](https://mlsanity.com/paper/2403.15482/) (2024)
- [Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation](https://mlsanity.com/paper/2403.13578/) (2024)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [VERVE: Template-based ReflectiVE Rewriting for MotiVational IntErviewing](https://mlsanity.com/paper/2311.08299/) (2023)
- [Modern Views of Machine Learning for Precision Psychiatry](https://mlsanity.com/paper/2204.01607/) (2022)
- [Towards Automated Real-time Evaluation in Text-based Counseling](https://mlsanity.com/paper/2203.03442/) (2022)
- [A Generic Review of Integrating Artificial Intelligence in Cognitive Behavioral Therapy](https://mlsanity.com/paper/2407.19422/) (2024)
- [An Automated Quality Evaluation Framework of Psychotherapy Conversations with Local Quality Estimates](https://mlsanity.com/paper/2106.07922/) (2021)
- [Fine-Tuning Open-Weight Language Models to Deliver Cognitive Behavioral Therapy for Depression: A Feasibility Study](https://mlsanity.com/paper/2412.00251/) (2024)
- [AI-Augmented LLMs Achieve Therapist-Level Responses in Motivational Interviewing](https://mlsanity.com/paper/2505.17380/) (2025)
- [Local dynamic mode of Cognitive Behavioral Therapy](https://mlsanity.com/paper/2205.09752/) (2022)
- [Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions](https://mlsanity.com/paper/2601.01237/) (2026)
- [Cognivia: A Cognitive Behavioral Therapy Copilot for Evidence-Based Mental Healthcare](https://mlsanity.com/paper/2607.25681/) (2026)

## Abstract (arXiv)

During a psychotherapy session, the counselor typically adopts techniques which are codified along specific dimensions (e.g., 'displays warmth and confidence', or 'attempts to set up collaboration') to facilitate the evaluation of the session. Those constructs, traditionally scored by trained human raters, reflect the complex nature of psychotherapy and highly depend on the context of the interaction. Recent advances in deep contextualized language models offer an avenue for accurate in-domain linguistic representations which can lead to robust recognition and scoring of such psychotherapy-relevant behavioral constructs, and support quality assurance and supervision. In this work, we propose a BERT-based model for automatic behavioral scoring of a specific type of psychotherapy, called Cognitive Behavioral Therapy (CBT), where prior work is limited to frequency-based language features and/or short text excerpts which do not capture the unique elements involved in a spontaneous long conversational interaction. The model focuses on the classification of therapy sessions with respect to the overall score achieved on the widely-used Cognitive Therapy Rating Scale (CTRS), but is trained in a multi-task manner in order to achieve higher interpretability. BERT-based representations are further augmented with available therapy metadata, providing relevant non-linguistic context and leading to consistent performance improvements. We train and evaluate our models on a set of 1,118 real-world therapy sessions, recorded and automatically transcribed. Our best model achieves an F1 score equal to 72.61% on the binary classification task of low vs. high total CTRS.