# Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization

arXiv: https://arxiv.org/abs/2501.16106 · HTML: https://mlsanity.com/paper/2501.16106/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 83 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 30 of 36)

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework](https://mlsanity.com/paper/2202.07123/) (2022)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques](https://mlsanity.com/paper/2005.01795/) (2020)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare](https://mlsanity.com/paper/2312.11541/) (2023)
- [MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysis](https://mlsanity.com/paper/2002.09283/) (2020)
- [Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Model](https://mlsanity.com/paper/2202.08210/) (2022)
- [D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat](https://mlsanity.com/paper/2205.11764/) (2022)
- [Topic Modeling Based Multi-modal Depression Detection](https://mlsanity.com/paper/1803.10384/) (2018)
- [PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents](https://mlsanity.com/paper/2402.12326/) (2024)
- [Improving the Generalizability of Depression Detection by Leveraging Clinical Questionnaires](https://mlsanity.com/paper/2204.10432/) (2022)
- [MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition](https://mlsanity.com/paper/2401.03429/) (2024)
- [Identifying Depressive Symptoms from Tweets: Figurative Language Enabled Multitask Learning Framework](https://mlsanity.com/paper/2011.06149/) (2020)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)
- [Symptom Identification for Interpretable Detection of Multiple Mental Disorders](https://mlsanity.com/paper/2205.11308/) (2022)
- [Multi-level Attention network using text, audio and video for Depression Prediction](https://mlsanity.com/paper/1909.01417/) (2019)
- [Self-supervised representations in speech-based depression detection](https://mlsanity.com/paper/2305.12263/) (2023)
- [When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection](https://mlsanity.com/paper/2402.13276/) (2024)
- [HiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection](https://mlsanity.com/paper/2408.03648/) (2024)
- [Towards Context-Aware Emotion Recognition Debiasing from a Causal Demystification Perspective via De-confounded Training](https://mlsanity.com/paper/2407.04963/) (2024)

## Built on by (most cited first, 1 of 1)

- [Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning](https://mlsanity.com/paper/2606.10796/) (2026)

## Abstract (arXiv)

Recent advances in multimodal depression recognition for clinical interviews (MDRC) have demonstrated the potential of AI systems by integrating textual, acoustic, and facial cues. However, existing methods pay limited attention to interpretability, thereby constraining reproducibility and clinician review. To address this, we introduce Explain-MDRC, an explainable MDRC framework that mirrors clinical workflows by generating structured symptom summaries from text and integrating them with nonverbal cues for recognition. Specifically, we construct Explain-DAIC, a dataset based on DAIC-WOZ and enriched with PHQ-8-aligned summary annotations, providing a foundation for developing models with built-in interpretability. We further propose PhqCML, a model that combines PHQ-8-aligned symptom summarization with PHQ-aware contrastive learning and summary-informed multimodal fusion. Automated metrics and expert evaluations show that Explain-MDRC improves recognition performance and provides more interpretable, clinician-readable intermediate evidence, suggesting a promising direction for transparent AI-assisted depression recognition research.