mlsanityData to 7 Oct 2026

Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

arXiv 2407.16804 ↗First posted 23 Jul 2024cs.LG cs.AI cs.CYcited by 7

Not on the map. Why?

Abstract (arXiv)

Multimodal machine learning (MML) is rapidly reshaping the way mental-health disorders are detected, characterized, and longitudinally monitored. Whereas early studies relied on isolated data streams -- such as speech, text, or wearable signals -- recent research has converged on architectures that integrate heterogeneous modalities to capture the rich, complex signatures of psychiatric conditions. This survey provides the first comprehensive, clinically grounded synthesis of MML for mental health. We (i) catalog 26 public datasets spanning audio, visual, physiological signals, and text modalities; (ii) systematically compare transformer, graph, and hybrid-based fusion strategies across 28 models, highlighting trends in representation learning and cross-modal alignment. Beyond summarizing current capabilities, we interrogate open challenges: data governance and privacy, demographic and intersectional fairness, evaluation explainability, and the complexity of mental health disorders in multimodal settings. By bridging methodological innovation with psychiatric utility, this survey aims to orient both ML researchers and mental-health practitioners toward the next generation of trustworthy, multimodal decision-support systems.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 17 of its 60 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

17 of 60 references.

References · 17

  1. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  2. 2021MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversationcited by 99
  3. 2019AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognitioncited by 87
  4. 2020MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysiscited by 38
  5. 2022COGMEN: COntextualized GNN based Multimodal Emotion recognitioNcited by 29
  6. 2022The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stresscited by 23
  7. 2023It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformerscited by 18
  8. 2021Graph Autoencoders for Embedding Learning in Brain Networks and Major Depressive Disorder Identificationcited by 15
  9. 2021A multimodal sensor dataset for continuous stress detection of nurses in a hospitalcited by 14
  10. 2022A Topic-Attentive Transformer-based Model For Multimodal Depression Detectioncited by 7
  11. 2022PTSD in the Wild: A Video Database for Studying Post-Traumatic Stress Disorder Recognition in Unconstrained Environmentscited by 6
  12. 2022WEMAC: Women and Emotion Multi-modal Affective Computing datasetcited by 5
5 more
  1. 2019See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Datacited by 4
  2. 2021MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Taskcited by 4
  3. 2022A multimodal stress detection dataset with facial expressions and physiological signalscited by 4
  4. 2024We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generationcited by 4
  5. 2024MOGAM: A Multimodal Object-oriented Graph Attention Model for Depression Detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.