Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges
Not on the map. Why?
Abstract (arXiv)
Multimodal machine learning (MML) is rapidly reshaping the way mental-health disorders are detected, characterized, and longitudinally monitored. Whereas early studies relied on isolated data streams -- such as speech, text, or wearable signals -- recent research has converged on architectures that integrate heterogeneous modalities to capture the rich, complex signatures of psychiatric conditions. This survey provides the first comprehensive, clinically grounded synthesis of MML for mental health. We (i) catalog 26 public datasets spanning audio, visual, physiological signals, and text modalities; (ii) systematically compare transformer, graph, and hybrid-based fusion strategies across 28 models, highlighting trends in representation learning and cross-modal alignment. Beyond summarizing current capabilities, we interrogate open challenges: data governance and privacy, demographic and intersectional fairness, evaluation explainability, and the complexity of mental health disorders in multimodal settings. By bridging methodological innovation with psychiatric utility, this survey aims to orient both ML researchers and mental-health practitioners toward the next generation of trustworthy, multimodal decision-support systems.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 17 of its 60 references that are in the map. First posted on arXiv 23 Jul 2024.
Builds on
17 of 60 references.
References · 17
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
- 2021MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversationcited by 99
- 2019AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognitioncited by 87
- 2020MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysiscited by 38
- 2022COGMEN: COntextualized GNN based Multimodal Emotion recognitioNcited by 29
- 2022The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stresscited by 23
- 2023It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformerscited by 18
- 2021Graph Autoencoders for Embedding Learning in Brain Networks and Major Depressive Disorder Identificationcited by 15
- 2021A multimodal sensor dataset for continuous stress detection of nurses in a hospitalcited by 14
- 2022A Topic-Attentive Transformer-based Model For Multimodal Depression Detectioncited by 7
- 2022PTSD in the Wild: A Video Database for Studying Post-Traumatic Stress Disorder Recognition in Unconstrained Environmentscited by 6
- 2022WEMAC: Women and Emotion Multi-modal Affective Computing datasetcited by 5
5 more
- 2019See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Datacited by 4
- 2021MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Taskcited by 4
- 2022A multimodal stress detection dataset with facial expressions and physiological signalscited by 4
- 2024We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generationcited by 4
- 2024MOGAM: A Multimodal Object-oriented Graph Attention Model for Depression Detectioncited by 2
Built on by
Citing papers per year: 2025: 5; 2026: 2.
7 papers.
- 2025A Comprehensive Review of Datasets for Clinical Mental Health AI Systemscited by 3
- 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunitiescited by 2
- 2025Unveiling the Landscape of Clinical Depression Assessment: From Behavioral Signatures to Psychiatric Reasoningcited by 2
- 2026A Comparative Study of Traditional Machine Learning, Deep Learning, and Large Language Models for Mental Health Forecasting using Smartphone Sensing Datacited by 1
- 2025Human Empathy as Encoder: AI-Assisted Depression Assessment in Special Educationcited by 0
- 2025SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosiscited by 0
- 2026From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Healthcited by 0
- 2026From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health21 Sep 2026
- 2026A Comparative Study of Traditional Machine Learning, Deep Learning, and Large Language Models for Mental Health Forecasting using Smartphone Sensing Data7 Jan 2026
- 2025SimClinician: A Multimodal Simulation Testbed for Reliable Psychologist AI Collaboration in Mental Health Diagnosis28 Nov 2025
- 2025A Comprehensive Review of Datasets for Clinical Mental Health AI Systems13 Aug 2025
- 2025Unveiling the Landscape of Clinical Depression Assessment: From Behavioral Signatures to Psychiatric Reasoning6 Aug 2025
- 2025Human Empathy as Encoder: AI-Assisted Depression Assessment in Special Education29 May 2025
- 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities1 Feb 2025