# MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for Dynamic Facial Expression Recognition

arXiv: https://arxiv.org/abs/2404.08433 · HTML: https://mlsanity.com/paper/2404.08433/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2206.04975/) (2022)
- [LOGO-Former: Local-Global Spatio-Temporal Transformer for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2305.03343/) (2023)
- [Anchor-Based Spatio-Temporal Attention 3D Convolutional Networks for Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/2012.10860/) (2020)

## Built on by (most cited first, 1 of 1)

- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)

## Abstract (arXiv)

Unlike typical video action recognition, Dynamic Facial Expression Recognition (DFER) does not involve distinct moving targets but relies on localized changes in facial muscles. Addressing this distinctive attribute, we propose a Multi-Scale Spatio-temporal CNN-Transformer network (MSSTNet). Our approach takes spatial features of different scales extracted by CNN and feeds them into a Multi-scale Embedding Layer (MELayer). The MELayer extracts multi-scale spatial information and encodes these features before sending them into a Temporal Transformer (T-Former). The T-Former simultaneously extracts temporal information while continually integrating multi-scale spatial information. This process culminates in the generation of multi-scale spatio-temporal features that are utilized for the final classification. Our method achieves state-of-the-art results on two in-the-wild datasets. Furthermore, a series of ablation experiments and visualizations provide further validation of our approach's proficiency in leveraging spatio-temporal information within DFER.