# Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

arXiv: https://arxiv.org/abs/2111.15266 · HTML: https://mlsanity.com/paper/2111.15266/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 106 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [Topic Modeling Based Multi-modal Depression Detection](https://mlsanity.com/paper/1803.10384/) (2018)
- [Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection](https://mlsanity.com/paper/1910.10334/) (2019)
- [Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution](https://mlsanity.com/paper/2004.09681/) (2020)
- [Measuring Depression Symptom Severity from Spoken Language and 3D Facial Expressions](https://mlsanity.com/paper/1811.08592/) (2018)
- [Video-based Facial Expression Recognition using Graph Convolutional Networks](https://mlsanity.com/paper/2010.13386/) (2020)
- [Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition](https://mlsanity.com/paper/2008.00859/) (2020)

## Built on by (most cited first, 10 of 10)

- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [Improving Depression estimation from facial videos with face alignment, training optimization and scheduling](https://mlsanity.com/paper/2212.06400/) (2022)
- [Fair Uncertainty Quantification for Depression Prediction](https://mlsanity.com/paper/2505.04931/) (2025)
- [Automatic Depression Assessment using Machine Learning: A Comprehensive Survey](https://mlsanity.com/paper/2506.18915/) (2025)
- [Domain-specific Learning of Multi-scale Facial Dynamics for Apparent Personality Traits Prediction](https://mlsanity.com/paper/2209.04148/) (2022)
- [ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognition](https://mlsanity.com/paper/2511.22188/) (2025)
- [Graph in Graph Neural Network](https://mlsanity.com/paper/2407.00696/) (2024)
- [On the Validity of Head Motion Patterns as Generalisable Depression Biomarkers](https://mlsanity.com/paper/2505.23427/) (2025)
- [EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning](https://mlsanity.com/paper/2604.16579/) (2026)
- [MA-DLE: Speech-based Automatic Depression Level Estimation via Memory Augmentation](https://mlsanity.com/paper/2606.11197/) (2026)

## Abstract (arXiv)

Video-based automatic depression analysis provides a fast, objective and repeatable self-assessment solution, which has been widely developed in recent years. While depression clues may be reflected by human facial behaviours of various temporal scales, most existing approaches either focused on modelling depression from short-term or video-level facial behaviours. In this sense, we propose a two-stage framework that models depression severity from multi-scale short-term and video-level facial behaviours. The short-term depressive behaviour modelling stage first deep learns depression-related facial behavioural features from multiple short temporal scales, where a Depression Feature Enhancement (DFE) module is proposed to enhance the depression-related clues for all temporal scales and remove non-depression noises. Then, the video-level depressive behaviour modelling stage proposes two novel graph encoding strategies, i.e., Sequential Graph Representation (SEG) and Spectral Graph Representation (SPG), to re-encode all short-term features of the target video into a video-level graph representation, summarizing depression-related multi-scale video-level temporal information. As a result, the produced graph representations predict depression severity using both short-term and long-term facial beahviour patterns. The experimental results on AVEC 2013 and AVEC 2014 datasets show that the proposed DFE module constantly enhanced the depression severity estimation performance for various CNN models while the SPG is superior than other video-level modelling methods. More importantly, the result achieved for the proposed two-stage framework shows its promising and solid performance compared to widely-used one-stage modelling approaches.