# Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ

arXiv: https://arxiv.org/abs/2407.19340 · HTML: https://mlsanity.com/paper/2407.19340/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 56 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction](https://mlsanity.com/paper/1702.04066/) (2017)
- [The Capability of Large Language Models to Measure Psychiatric Functioning](https://mlsanity.com/paper/2308.01834/) (2023)
- [Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis](https://mlsanity.com/paper/2206.06367/) (2022)
- [Multi-modal Depression Estimation based on Sub-attentional Fusion](https://mlsanity.com/paper/2207.06180/) (2022)
- [Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues](https://mlsanity.com/paper/2401.02746/) (2024)
- [Predicting Depression Severity by Multi-Modal Feature Engineering and Fusion](https://mlsanity.com/paper/1711.11155/) (2017)

## Built on by (most cited first, 4 of 4)

- [Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance](https://mlsanity.com/paper/2412.10417/) (2024)
- [CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Dataset](https://mlsanity.com/paper/2506.16385/) (2025)
- [Leveraging Embedding Techniques in Multimodal Machine Learning for Mental Illness Assessment](https://mlsanity.com/paper/2504.01767/) (2025)
- [A Multi-Agent Audit Framework for High-Stakes Reasoning: Evaluation and Interpretability in Clinical Mental Health Screening](https://mlsanity.com/paper/2606.21123/) (2026)

## Abstract (arXiv)

Major Depressive Disorder (MDD) is a pervasive mental health condition that affects 300 million people worldwide. This work presents a novel, BiLSTM-based tri-modal model-level fusion architecture for the binary classification of depression from clinical interview recordings. The proposed architecture incorporates Mel Frequency Cepstral Coefficients, Facial Action Units, and uses a two-shot learning based GPT-4 model to process text data. This is the first work to incorporate large language models into a multi-modal architecture for this task. It achieves impressive results on the DAIC-WOZ AVEC 2016 Challenge cross-validation split and Leave-One-Subject-Out cross-validation split, surpassing all baseline models and multiple state-of-the-art models. In Leave-One-Subject-Out testing, it achieves an accuracy of 91.01%, an F1-Score of 85.95%, a precision of 80%, and a recall of 92.86%.