mlsanityData to 7 Oct 2026

Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues

arXiv 2401.02746 ↗First posted 5 Jan 2024cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 30 of its 95 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

30 of 95 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  7. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  9. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  10. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 175
  11. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  12. 2020ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variationcited by 141
18 more
  1. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  2. 2020BlazePose: On-device Real-time Body Pose trackingcited by 112
  3. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  4. 2019AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognitioncited by 87
  5. 2017Depression and Self-Harm Risk Assessment in Online Forumscited by 81
  6. 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 76
  7. 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
  8. 2021Deep Learning for Suicide and Depression Identification with Unsupervised Label Correctioncited by 45
  9. 2016Instagram photos reveal predictive markers of depressioncited by 25
  10. 2017Multi-Task Learning for Mental Health using Social Media Textcited by 25
  11. 2021Deep Learning for Depression Recognition with Audiovisual Cues: A Reviewcited by 24
  12. 2020Gender Bias in Depression Detection Using Audio Featurescited by 21
  13. 2021Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directionscited by 21
  14. 2019Multi-level Attention network using text, audio and video for Depression Predictioncited by 19
  15. 2023It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformerscited by 18
  16. 2022Multi-modal Depression Estimation based on Sub-attentional Fusioncited by 15
  17. 2023Real-time Multi-person Eyeblink Detection in the Wild for Untrimmed Videocited by 3
  18. 2021Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 2.

11 papers.

  1. 2024Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosiscited by 8
  2. 2024On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlookcited by 6
  3. 2024Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalitiescited by 5
  4. 2024Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZcited by 4
  5. 2025Automatic Depression Assessment using Machine Learning: A Comprehensive Surveycited by 1
  6. 2025BigTokDetect: A Clinically-Informed Vision-Language Modeling Framework for Detecting Pro-Bigorexia Videos on TikTokcited by 1
  7. 2025FairSSL: Fair Multimodal Self-Supervised Learningcited by 1
  8. 2025Database-Agnostic Gait Enrollment using SetTransformerscited by 0
  9. 2025MoME: Estimating Psychological Traits from Gait with Multi-Stage Mixture of Movement Expertscited by 0
  10. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  11. 2026TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairmentcited by 0