# Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech Samples

arXiv: https://arxiv.org/abs/1909.01067 · HTML: https://mlsanity.com/paper/1909.01067/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 3 of 3)

- [psifx -- Psychological and Social Interactions Feature Extraction Package](https://mlsanity.com/paper/2407.10266/) (2024)
- [Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation](https://mlsanity.com/paper/2604.16615/) (2026)
- [Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning](https://mlsanity.com/paper/2604.16657/) (2026)

## Abstract (arXiv)

Key features of mental illnesses are reflected in speech. Our research focuses on designing a multimodal deep learning structure that automatically extracts salient features from recorded speech samples for predicting various mental disorders including depression, bipolar, and schizophrenia. We adopt a variety of pre-trained models to extract embeddings from both audio and text segments. We use several state-of-the-art embedding techniques including BERT, FastText, and Doc2VecC for the text representation learning and WaveNet and VGG-ish models for audio encoding. We also leverage huge auxiliary emotion-labeled text and audio corpora to train emotion-specific embeddings and use transfer learning in order to address the problem of insufficient annotated multimodal data available. All these embeddings are then combined into a joint representation in a multimodal fusion layer and finally a recurrent neural network is used to predict the mental disorder. Our results show that mental disorders can be predicted with acceptable accuracy through multimodal analysis of clinical interviews.