# Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields

arXiv: https://arxiv.org/abs/1703.06995 · HTML: https://mlsanity.com/paper/1703.06995/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2017.

## Built on by (most cited first, 6 of 6)

- [Detection and Localization of Facial Expression Manipulations](https://mlsanity.com/paper/2103.08134/) (2021)
- [Critically examining the Domain Generalizability of Facial Expression Recognition models](https://mlsanity.com/paper/2106.15453/) (2021)
- [How far generated data can impact Neural Networks performance?](https://mlsanity.com/paper/2303.15223/) (2023)
- [On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study](https://mlsanity.com/paper/2104.02653/) (2021)
- [What happens in Face during a facial expression? Using data mining techniques to analyze facial expression motion vectors](https://mlsanity.com/paper/2109.05457/) (2021)
- [Data Augmentation and Transfer Learning Approaches Applied to Facial Expressions Recognition](https://mlsanity.com/paper/2402.09982/) (2024)

## Abstract (arXiv)

Automated Facial Expression Recognition (FER) has been a challenging task for decades. Many of the existing works use hand-crafted features such as LBP, HOG, LPQ, and Histogram of Optical Flow (HOF) combined with classifiers such as Support Vector Machines for expression recognition. These methods often require rigorous hyperparameter tuning to achieve good results. Recently Deep Neural Networks (DNN) have shown to outperform traditional methods in visual object recognition. In this paper, we propose a two-part network consisting of a DNN-based architecture followed by a Conditional Random Field (CRF) module for facial expression recognition in videos. The first part captures the spatial relation within facial images using convolutional layers followed by three Inception-ResNet modules and two fully-connected layers. To capture the temporal relation between the image frames, we use linear chain CRF in the second part of our network. We evaluate our proposed network on three publicly available databases, viz. CK+, MMI, and FERA. Experiments are performed in subject-independent and cross-database manners. Our experimental results show that cascading the deep network architecture with the CRF module considerably increases the recognition of facial expressions in videos and in particular it outperforms the state-of-the-art methods in the cross-database experiments and yields comparable results in the subject-independent experiments.