# Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance

arXiv: https://arxiv.org/abs/2412.10417 · HTML: https://mlsanity.com/paper/2412.10417/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [The Capability of Large Language Models to Measure Psychiatric Functioning](https://mlsanity.com/paper/2308.01834/) (2023)
- [Multi-modal Depression Estimation based on Sub-attentional Fusion](https://mlsanity.com/paper/2207.06180/) (2022)
- [DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews](https://mlsanity.com/paper/2404.14463/) (2024)
- [Depression Severity Estimation from Multiple Modalities](https://mlsanity.com/paper/1711.06095/) (2017)
- [A Comprehensive Evaluation of Large Language Models on Mental Illnesses](https://mlsanity.com/paper/2409.15687/) (2024)
- [Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities](https://mlsanity.com/paper/2407.06125/) (2024)
- [Assessing ML Classification Algorithms and NLP Techniques for Depression Detection: An Experimental Case Study](https://mlsanity.com/paper/2404.04284/) (2024)
- [Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ](https://mlsanity.com/paper/2407.19340/) (2024)
- [Confidence Estimation for Automatic Detection of Depression and Alzheimer's Disease Based on Clinical Interviews](https://mlsanity.com/paper/2407.19984/) (2024)
- [Density Adaptive Attention-based Speech Network: Enhancing Feature Understanding for Mental Health Disorders](https://mlsanity.com/paper/2409.00391/) (2024)
- [Integration of Text and Graph-based Features for Detecting Mental Health Disorders from Voice](https://mlsanity.com/paper/2205.07006/) (2022)

## Built on by (most cited first, 4 of 4)

- [PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry](https://mlsanity.com/paper/2509.09711/) (2025)
- [Predicting Depression in Screening Interviews from Interactive Multi-Theme Collaboration](https://mlsanity.com/paper/2502.12204/) (2025)
- [Leveraging Embedding Techniques in Multimodal Machine Learning for Mental Illness Assessment](https://mlsanity.com/paper/2504.01767/) (2025)
- [When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews](https://mlsanity.com/paper/2603.24651/) (2026)

## Abstract (arXiv)

Mental health disorders are increasingly prevalent worldwide, creating an urgent need for innovative tools to support early diagnosis and intervention. This study explores the potential of Large Language Models (LLMs) in multimodal mental health diagnostics, specifically for detecting depression and Post Traumatic Stress Disorder through text and audio modalities. Using the E-DAIC dataset, we compare text and audio modalities to investigate whether LLMs can perform equally well or better with audio inputs. We further examine the integration of both modalities to determine if this can enhance diagnostic accuracy, which generally results in improved performance metrics. Our analysis specifically utilizes custom-formulated metrics; Modal Superiority Score and Disagreement Resolvement Score to evaluate how combined modalities influence model performance. The Gemini 1.5 Pro model achieves the highest scores in binary depression classification when using the combined modality, with an F1 score of 0.67 and a Balanced Accuracy (BA) of 77.4%, assessed across the full dataset. These results represent an increase of 3.1% over its performance with the text modality and 2.7% over the audio modality, highlighting the effectiveness of integrating modalities to enhance diagnostic accuracy. Notably, all results are obtained in zero-shot inferring, highlighting the robustness of the models without requiring task-specific fine-tuning. To explore the impact of different configurations on model performance, we conduct binary, severity, and multiclass tasks using both zero-shot and few-shot prompts, examining the effects of prompt variations on performance. The results reveal that models such as Gemini 1.5 Pro in text and audio modalities, and GPT-4o mini in the text modality, often surpass other models in balanced accuracy and F1 scores across multiple tasks.