# Leveraging Embedding Techniques in Multimodal Machine Learning for Mental Illness Assessment

arXiv: https://arxiv.org/abs/2504.01767 · HTML: https://mlsanity.com/paper/2504.01767/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 8 of 8)

- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews](https://mlsanity.com/paper/2404.14463/) (2024)
- [A Comprehensive Evaluation of Large Language Models on Mental Illnesses](https://mlsanity.com/paper/2409.15687/) (2024)
- [Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities](https://mlsanity.com/paper/2407.06125/) (2024)
- [Assessing ML Classification Algorithms and NLP Techniques for Depression Detection: An Experimental Case Study](https://mlsanity.com/paper/2404.04284/) (2024)
- [Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ](https://mlsanity.com/paper/2407.19340/) (2024)
- [Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance](https://mlsanity.com/paper/2412.10417/) (2024)

## Built on by (most cited first, 1 of 1)

- [PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry](https://mlsanity.com/paper/2509.09711/) (2025)

## Abstract (arXiv)

The increasing global prevalence of mental disorders, such as depression and PTSD, requires objective and scalable diagnostic tools. Traditional clinical assessments often face limitations in accessibility, objectivity, and consistency. This paper investigates the potential of multimodal machine learning to address these challenges, leveraging the complementary information available in text, audio, and video data. Our approach involves a comprehensive analysis of various data preprocessing techniques, including novel chunking and utterance-based formatting strategies. We systematically evaluate a range of state-of-the-art embedding models for each modality and employ Convolutional Neural Networks (CNNs) and Bidirectional LSTM Networks (BiLSTMs) for feature extraction. We explore data-level, feature-level, and decision-level fusion techniques, including a novel integration of Large Language Model (LLM) predictions. We also investigate the impact of replacing Multilayer Perceptron classifiers with Support Vector Machines. We extend our analysis to severity prediction using PHQ-8 and PCL-C scores and multi-class classification (considering co-occurring conditions). Our results demonstrate that utterance-based chunking significantly improves performance, particularly for text and audio modalities. Decision-level fusion, incorporating LLM predictions, achieves the highest accuracy, with a balanced accuracy of 94.8% for depression and 96.2% for PTSD detection. The combination of CNN-BiLSTM architectures with utterance-level chunking, coupled with the integration of external LLM, provides a powerful and nuanced approach to the detection and assessment of mental health conditions. Our findings highlight the potential of MMML for developing more accurate, accessible, and personalized mental healthcare tools.