# Do Deep Neural Networks Forget Facial Action Units? -- Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition

arXiv: https://arxiv.org/abs/2104.07389 · HTML: https://mlsanity.com/paper/2104.07389/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 12 of 12)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)
- [Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?](https://mlsanity.com/paper/1510.02969/) (2015)
- [Fusing Deep Learned and Hand-Crafted Features of Appearance, Shape, and Dynamics for Automatic Pain Estimation](https://mlsanity.com/paper/1701.04540/) (2017)
- [Neonatal Pain Expression Recognition Using Transfer Learning](https://mlsanity.com/paper/1807.01631/) (2018)

## Built on by (most cited first, 3 of 3)

- [Faces of Experimental Pain: Transferability of Deep Learned Heat Pain Features to Electrical Pain](https://mlsanity.com/paper/2406.11808/) (2024)
- [Towards a Deep Learning Pain-Level Detection Deployment at UAE for Patient-Centric-Pain Management and Diagnosis Support: Framework and Performance Evaluation](https://mlsanity.com/paper/2303.08273/) (2023)
- [Socially Interactive Agents for Robotic Neurorehabilitation Training: Conceptualization and Proof-of-concept Study](https://mlsanity.com/paper/2406.12035/) (2024)

## Abstract (arXiv)

In this paper, we present a process to investigate the effects of transfer learning for automatic facial expression recognition from emotions to pain. To this end, we first train a VGG16 convolutional neural network to automatically discern between eight categorical emotions. We then fine-tune successively larger parts of this network to learn suitable representations for the task of automatic pain recognition. Subsequently, we apply those fine-tuned representations again to the original task of emotion recognition to further investigate the differences in performance between the models. In the second step, we use Layer-wise Relevance Propagation to analyze predictions of the model that have been predicted correctly previously but are now wrongly classified. Based on this analysis, we rely on the visual inspection of a human observer to generate hypotheses about what has been forgotten by the model. Finally, we test those hypotheses quantitatively utilizing concept embedding analysis methods. Our results show that the network, which was fully fine-tuned for pain recognition, indeed payed less attention to two action units that are relevant for expression recognition but not for pain recognition.