# Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities

arXiv: https://arxiv.org/abs/2407.06125 · HTML: https://mlsanity.com/paper/2407.06125/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation](https://mlsanity.com/paper/2207.14087/) (2022)
- [Multi-level Attention network using text, audio and video for Depression Prediction](https://mlsanity.com/paper/1909.01417/) (2019)
- [MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting](https://mlsanity.com/paper/2404.08704/) (2024)
- [Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks](https://mlsanity.com/paper/2404.13099/) (2024)
- [Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues](https://mlsanity.com/paper/2401.02746/) (2024)
- [Prediction of Depression Severity Based on the Prosodic and Semantic Features with Bidirectional LSTM and Time Distributed CNN](https://mlsanity.com/paper/2202.12456/) (2022)
- [Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion](https://mlsanity.com/paper/2301.00254/) (2022)
- [Advancements in Scientific Controllable Text Generation Methods](https://mlsanity.com/paper/2307.05538/) (2023)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)
- [Context-Enhanced Language Models for Generating Multi-Paper Citations](https://mlsanity.com/paper/2404.13865/) (2024)
- [KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models](https://mlsanity.com/paper/2404.09763/) (2024)

## Built on by (most cited first, 5 of 5)

- [Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance](https://mlsanity.com/paper/2412.10417/) (2024)
- [Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge](https://mlsanity.com/paper/2505.22863/) (2025)
- [Leveraging Embedding Techniques in Multimodal Machine Learning for Mental Illness Assessment](https://mlsanity.com/paper/2504.01767/) (2025)
- [Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models](https://mlsanity.com/paper/2412.01353/) (2024)
- [Using Large Language Models to Measure Symptom Severity in Patients At Risk for Schizophrenia](https://mlsanity.com/paper/2508.10226/) (2025)

## Abstract (arXiv)

Depression has proven to be a significant public health issue, profoundly affecting the psychological well-being of individuals. If it remains undiagnosed, depression can lead to severe health issues, which can manifest physically and even lead to suicide. Generally, Diagnosing depression or any other mental disorder involves conducting semi-structured interviews alongside supplementary questionnaires, including variants of the Patient Health Questionnaire (PHQ) by Clinicians and mental health professionals. This approach places significant reliance on the experience and judgment of trained physicians, making the diagnosis susceptible to personal biases. Given that the underlying mechanisms causing depression are still being actively researched, physicians often face challenges in diagnosing and treating the condition, particularly in its early stages of clinical presentation. Recently, significant strides have been made in Artificial neural computing to solve problems involving text, image, and speech in various domains. Our analysis has aimed to leverage these state-of-the-art (SOTA) models in our experiments to achieve optimal outcomes leveraging multiple modalities. The experiments were performed on the Extended Distress Analysis Interview Corpus Wizard of Oz dataset (E-DAIC) corpus presented in the Audio/Visual Emotion Challenge (AVEC) 2019 Challenge. The proposed solutions demonstrate better results achieved by Proprietary and Open-source Large Language Models (LLMs), which achieved a Root Mean Square Error (RMSE) score of 3.98 on Textual Modality, beating the AVEC 2019 challenge baseline results and current SOTA regression analysis architectures. Additionally, the proposed solution achieved an accuracy of 71.43% in the classification task. The paper also includes a novel audio-visual multi-modal network that predicts PHQ-8 scores with an RMSE of 6.51.