# Seeing and hearing what has not been said; A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

arXiv: https://arxiv.org/abs/2309.14398 · HTML: https://mlsanity.com/paper/2309.14398/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [EmbraceNet for Activity: A Deep Multimodal Fusion Architecture for Activity Recognition](https://mlsanity.com/paper/2004.13918/) (2020)
- [Feature Fusion Strategies for End-to-End Evaluation of Cognitive Behavior Therapy Sessions](https://mlsanity.com/paper/2005.07809/) (2020)

## Built on by (most cited first, 3 of 3)

- [EMMI -- Empathic Multimodal Motivational Interviews Dataset: Analyses and Annotations](https://mlsanity.com/paper/2406.16478/) (2024)
- [Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems](https://mlsanity.com/paper/2412.04492/) (2024)
- [Leveraging Multimodal Self-Consistency Reasoning in Coding Motivational Interviewing for Alcohol Use Reduction](https://mlsanity.com/paper/2605.12987/) (2026)

## Abstract (arXiv)

Motivational Interviewing (MI) is an approach to therapy that emphasizes collaboration and encourages behavioral change. To evaluate the quality of an MI conversation, client utterances can be classified using the MISC code as either change talk, sustain talk, or follow/neutral talk. The proportion of change talk in a MI conversation is positively correlated with therapy outcomes, making accurate classification of client utterances essential. In this paper, we present a classifier that accurately distinguishes between the three MISC classes (change talk, sustain talk, and follow/neutral talk) leveraging multimodal features such as text, prosody, facial expressivity, and body expressivity. To train our model, we perform annotations on the publicly available AnnoMI dataset to collect multimodal information, including text, audio, facial expressivity, and body expressivity. Furthermore, we identify the most important modalities in the decision-making process, providing valuable insights into the interplay of different modalities during a MI conversation.