# Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

arXiv: https://arxiv.org/abs/2407.16804 · HTML: https://mlsanity.com/paper/2407.16804/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 60 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysis](https://mlsanity.com/paper/2002.09283/) (2020)
- [COGMEN: COntextualized GNN based Multimodal Emotion recognitioN](https://mlsanity.com/paper/2205.02455/) (2022)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers](https://mlsanity.com/paper/2301.05453/) (2023)
- [Graph Autoencoders for Embedding Learning in Brain Networks and Major Depressive Disorder Identification](https://mlsanity.com/paper/2107.12838/) (2021)
- [A multimodal sensor dataset for continuous stress detection of nurses in a hospital](https://mlsanity.com/paper/2108.07689/) (2021)
- [A Topic-Attentive Transformer-based Model For Multimodal Depression Detection](https://mlsanity.com/paper/2206.13256/) (2022)
- [PTSD in the Wild: A Video Database for Studying Post-Traumatic Stress Disorder Recognition in Unconstrained Environments](https://mlsanity.com/paper/2209.14085/) (2022)
- [WEMAC: Women and Emotion Multi-modal Affective Computing dataset](https://mlsanity.com/paper/2203.00456/) (2022)
- [See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Data](https://mlsanity.com/paper/1912.01131/) (2019)
- [MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task](https://mlsanity.com/paper/2105.08146/) (2021)
- [A multimodal stress detection dataset with facial expressions and physiological signals](https://mlsanity.com/paper/2209.13542/) (2022)
- [We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generation](https://mlsanity.com/paper/2406.10561/) (2024)
- [MOGAM: A Multimodal Object-oriented Graph Attention Model for Depression Detection](https://mlsanity.com/paper/2403.15485/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Review of Datasets for Clinical Mental Health AI Systems](https://mlsanity.com/paper/2508.09809/) (2025)
- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)
- [Unveiling the Landscape of Clinical Depression Assessment: From Behavioral Signatures to Psychiatric Reasoning](https://mlsanity.com/paper/2508.04531/) (2025)
- [A Comparative Study of Traditional Machine Learning, Deep Learning, and Large Language Models for Mental Health Forecasting using Smartphone Sensing Data](https://mlsanity.com/paper/2601.03603/) (2026)
- [Human Empathy as Encoder: AI-Assisted Depression Assessment in Special Education](https://mlsanity.com/paper/2505.23631/) (2025)
- [SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosis](https://mlsanity.com/paper/2512.08953/) (2025)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

Multimodal machine learning (MML) is rapidly reshaping the way mental-health disorders are detected, characterized, and longitudinally monitored. Whereas early studies relied on isolated data streams -- such as speech, text, or wearable signals -- recent research has converged on architectures that integrate heterogeneous modalities to capture the rich, complex signatures of psychiatric conditions. This survey provides the first comprehensive, clinically grounded synthesis of MML for mental health. We (i) catalog 26 public datasets spanning audio, visual, physiological signals, and text modalities; (ii) systematically compare transformer, graph, and hybrid-based fusion strategies across 28 models, highlighting trends in representation learning and cross-modal alignment. Beyond summarizing current capabilities, we interrogate open challenges: data governance and privacy, demographic and intersectional fairness, evaluation explainability, and the complexity of mental health disorders in multimodal settings. By bridging methodological innovation with psychiatric utility, this survey aims to orient both ML researchers and mental-health practitioners toward the next generation of trustworthy, multimodal decision-support systems.