# Toward Knowledge-Driven Speech-Based Models of Depression: Leveraging Spectrotemporal Variations in Speech Vowels

arXiv: https://arxiv.org/abs/2210.02527 · HTML: https://mlsanity.com/paper/2210.02527/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 6 of 6)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech](https://mlsanity.com/paper/2203.03812/) (2022)
- [AudVowelConsNet: A Phoneme-Level Based Deep CNN Architecture for Clinical Depression Diagnosis](https://mlsanity.com/paper/2010.16201/) (2020)

## Built on by (most cited first, 3 of 3)

- [A knowledge-driven vowel-based approach of depression classification from speech using data augmentation](https://mlsanity.com/paper/2210.15261/) (2022)
- [Robust and Explainable Depression Identification from Speech Using Vowel-Based Ensemble Learning Approaches](https://mlsanity.com/paper/2410.18298/) (2024)
- [The Role of Explainable AI in Revolutionizing Human Health Monitoring: A Review](https://mlsanity.com/paper/2409.07347/) (2024)

## Abstract (arXiv)

Psychomotor retardation associated with depression has been linked with tangible differences in vowel production. This paper investigates a knowledge-driven machine learning (ML) method that integrates spectrotemporal information of speech at the vowel-level to identify the depression. Low-level speech descriptors are learned by a convolutional neural network (CNN) that is trained for vowel classification. The temporal evolution of those low-level descriptors is modeled at the high-level within and across utterances via a long short-term memory (LSTM) model that takes the final depression decision. A modified version of the Local Interpretable Model-agnostic Explanations (LIME) is further used to identify the impact of the low-level spectrotemporal vowel variation on the decisions and observe the high-level temporal change of the depression likelihood. The proposed method outperforms baselines that model the spectrotemporal information in speech without integrating the vowel-based information, as well as ML models trained with conventional prosodic and spectrotemporal features. The conducted explainability analysis indicates that spectrotemporal information corresponding to non-vowel segments less important than the vowel-based information. Explainability of the high-level information capturing the segment-by-segment decisions is further inspected for participants with and without depression. The findings from this work can provide the foundation toward knowledge-driven interpretable decision-support systems that can assist clinicians to better understand fine-grain temporal changes in speech data, ultimately augmenting mental health diagnosis and care.