# Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?

arXiv: https://arxiv.org/abs/1510.02969 · HTML: https://mlsanity.com/paper/1510.02969/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2015.

## Built on by (most cited first, 19 of 19)

- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)
- [Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition](https://mlsanity.com/paper/2303.04896/) (2023)
- [Addressing Racial Bias in Facial Emotion Recognition](https://mlsanity.com/paper/2308.04674/) (2023)
- [An Ensemble with Shared Representations Based on Convolutional Networks for Continually Learning Facial Expressions](https://mlsanity.com/paper/2103.03934/) (2021)
- [Do Deep Neural Networks Forget Facial Action Units? -- Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition](https://mlsanity.com/paper/2104.07389/) (2021)
- [Efficient Neural Architecture Search for Emotion Recognition](https://mlsanity.com/paper/2303.13653/) (2023)
- [Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI](https://mlsanity.com/paper/2401.11835/) (2024)
- [Landmark-Aware and Part-based Ensemble Transfer Learning Network for Facial Expression Recognition from Static images](https://mlsanity.com/paper/2104.11274/) (2021)
- [Deep Learning Models for Automated Classification of Dog Emotional States from Facial Expressions](https://mlsanity.com/paper/2206.05619/) (2022)
- [Scalar Invariant Networks with Zero Bias](https://mlsanity.com/paper/2211.08486/) (2022)
- [A Sub-Layered Hierarchical Pyramidal Neural Architecture for Facial Expression Recognition](https://mlsanity.com/paper/2103.12362/) (2021)
- [Domain Adaptation for Facial Expression Classifier via Domain Discrimination and Gradient Reversal](https://mlsanity.com/paper/2106.01467/) (2021)
- [Exploring Adversarial Learning for Deep Semi-Supervised Facial Action Unit Recognition](https://mlsanity.com/paper/2106.02258/) (2021)
- [Cross-Centroid Ripple Pattern for Facial Expression Recognition](https://mlsanity.com/paper/2201.05958/) (2022)
- [RARITYNet: Rarity Guided Affective Emotion Learning Framework](https://mlsanity.com/paper/2205.08595/) (2022)
- [Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches](https://mlsanity.com/paper/2307.06625/) (2023)
- [InFER: A Multi-Ethnic Indian Facial Expression Recognition Dataset](https://mlsanity.com/paper/2310.00287/) (2023)

## Abstract (arXiv)

Despite being the appearance-based classifier of choice in recent years, relatively few works have examined how much convolutional neural networks (CNNs) can improve performance on accepted expression recognition benchmarks and, more importantly, examine what it is they actually learn. In this work, not only do we show that CNNs can achieve strong performance, but we also introduce an approach to decipher which portions of the face influence the CNN's predictions. First, we train a zero-bias CNN on facial expression data and achieve, to our knowledge, state-of-the-art performance on two expression recognition benchmarks: the extended Cohn-Kanade (CK+) dataset and the Toronto Face Dataset (TFD). We then qualitatively analyze the network by visualizing the spatial patterns that maximally excite different neurons in the convolutional layers and show how they resemble Facial Action Units (FAUs). Finally, we use the FAU labels provided in the CK+ dataset to verify that the FAUs observed in our filter visualizations indeed align with the subject's facial movements.