# Facial Expression Analysis Using Decomposed Multiscale Spatiotemporal Networks

arXiv: https://arxiv.org/abs/2203.11111 · HTML: https://mlsanity.com/paper/2203.11111/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 84 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Recurrent Convolutional Neural Network Regression for Continuous Pain Intensity Estimation in Video](https://mlsanity.com/paper/1605.00894/) (2016)
- [Deep Weakly-Supervised Domain Adaptation for Pain Localization in Videos](https://mlsanity.com/paper/1910.08173/) (2019)
- [Deep Domain Adaptation for Ordinal Regression of Pain Intensity Estimation Using Weakly-Labelled Videos](https://mlsanity.com/paper/2008.06392/) (2020)
- [Automatic Detection of ADHD and ASD from Expressive Behaviour in RGBD Data](https://mlsanity.com/paper/1612.02374/) (2016)
- [Multi-Instance Dynamic Ordinal Random Fields for Weakly-supervised Facial Behavior Analysis](https://mlsanity.com/paper/1803.00907/) (2018)

## Built on by (most cited first, 4 of 4)

- [Depression Recognition using Remote Photoplethysmography from Facial Videos](https://mlsanity.com/paper/2206.04399/) (2022)
- [FacEnhance: Facial Expression Enhancing with Recurrent DDPMs](https://mlsanity.com/paper/2406.09040/) (2024)
- [Trajectory-guided Motion Perception for Facial Expression Quality Assessment in Neurological Disorders](https://mlsanity.com/paper/2504.09530/) (2025)
- [TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment](https://mlsanity.com/paper/2606.26942/) (2026)

## Abstract (arXiv)

Video-based analysis of facial expressions has been increasingly applied to infer health states of individuals, such as depression and pain. Among the existing approaches, deep learning models composed of structures for multiscale spatiotemporal processing have shown strong potential for encoding facial dynamics. However, such models have high computational complexity, making for a difficult deployment of these solutions. To address this issue, we introduce a new technique to decompose the extraction of multiscale spatiotemporal features. Particularly, a building block structure called Decomposed Multiscale Spatiotemporal Network (DMSN) is presented along with three variants: DMSN-A, DMSN-B, and DMSN-C blocks. The DMSN-A block generates multiscale representations by analyzing spatiotemporal features at multiple temporal ranges, while the DMSN-B block analyzes spatiotemporal features at multiple ranges, and the DMSN-C block analyzes spatiotemporal features at multiple spatial sizes. Using these variants, we design our DMSN architecture which has the ability to explore a variety of multiscale spatiotemporal features, favoring the adaptation to different facial behaviors. Our extensive experiments on challenging datasets show that the DMSN-C block is effective for depression detection, whereas the DMSN-A block is efficient for pain estimation. Results also indicate that our DMSN architecture provides a cost-effective solution for expressions that range from fewer facial variations over time, as in depression detection, to greater variations, as in pain estimation.