# Measuring Depression Symptom Severity from Spoken Language and 3D Facial Expressions

arXiv: https://arxiv.org/abs/1811.08592 · HTML: https://mlsanity.com/paper/1811.08592/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2018.

## Built on by (most cited first, 21 of 21)

- [Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Model](https://mlsanity.com/paper/2202.08210/) (2022)
- [Multi-modal Depression Estimation based on Sub-attentional Fusion](https://mlsanity.com/paper/2207.06180/) (2022)
- [Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation](https://mlsanity.com/paper/2111.15266/) (2021)
- [Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge](https://mlsanity.com/paper/2303.09158/) (2023)
- [DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation](https://mlsanity.com/paper/2408.06044/) (2024)
- [Automatic Depression Detection via Learning and Fusing Features from Visual Cues](https://mlsanity.com/paper/2203.00304/) (2022)
- [Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation](https://mlsanity.com/paper/2106.13393/) (2021)
- [We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generation](https://mlsanity.com/paper/2406.10561/) (2024)
- [Predicting Mood Disorder Symptoms with Remotely Collected Videos Using an Interpretable Multimodal Dynamic Attention Fusion Network](https://mlsanity.com/paper/2109.03029/) (2021)
- [Neural Architecture Searching for Facial Attributes-based Depression Recognition](https://mlsanity.com/paper/2201.09799/) (2022)
- [OpticalDR: A Deep Optical Imaging Model for Privacy-Protective Depression Recognition](https://mlsanity.com/paper/2402.18786/) (2024)
- [Automatic Depression Assessment using Machine Learning: A Comprehensive Survey](https://mlsanity.com/paper/2506.18915/) (2025)
- [It Hears, It Sees too: Multi-Modal LLM for Depression Detection By Integrating Visual Understanding into Audio Language Models](https://mlsanity.com/paper/2511.19877/) (2025)
- [Multi-Task Learning for Depression Detection in Dialogs](https://mlsanity.com/paper/2208.10250/) (2022)
- [Cost-effective Models for Detecting Depression from Speech](https://mlsanity.com/paper/2302.09214/) (2023)
- [Using Mobile Data and Deep Models to Assess Auditory Verbal Hallucinations](https://mlsanity.com/paper/2304.11049/) (2023)
- [When and How to Integrate Multimodal Large Language Models in College Psychotherapy: Perspectives from Multi-stakeholders](https://mlsanity.com/paper/2502.00229/) (2025)
- [Lightweight Models for Emotional Analysis in Video](https://mlsanity.com/paper/2503.10530/) (2025)
- [Entropy-Dominated Temporal Vocal Dynamics as Digital Biomarkers for Depression Detection](https://mlsanity.com/paper/2604.26998/) (2026)
- [Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis](https://mlsanity.com/paper/2606.25606/) (2026)
- [Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training](https://mlsanity.com/paper/2607.22794/) (2026)

## Abstract (arXiv)

With more than 300 million people depressed worldwide, depression is a global problem. Due to access barriers such as social stigma, cost, and treatment availability, 60% of mentally-ill adults do not receive any mental health services. Effective and efficient diagnosis relies on detecting clinical symptoms of depression. Automatic detection of depressive symptoms would potentially improve diagnostic accuracy and availability, leading to faster intervention. In this work, we present a machine learning method for measuring the severity of depressive symptoms. Our multi-modal method uses 3D facial expressions and spoken language, commonly available from modern cell phones. It demonstrates an average error of 3.67 points (15.3% relative) on the clinically-validated Patient Health Questionnaire (PHQ) scale. For detecting major depressive disorder, our model demonstrates 83.3% sensitivity and 82.6% specificity. Overall, this paper shows how speech recognition, computer vision, and natural language processing can be combined to assist mental health patients and practitioners. This technology could be deployed to cell phones worldwide and facilitate low-cost universal access to mental health care.