# AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection

arXiv: https://arxiv.org/abs/1812.05788 · HTML: https://mlsanity.com/paper/1812.05788/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 14 of 14)

- [Feature Representation Learning with Adaptive Displacement Generation and Transformer Fusion for Micro-Expression Recognition](https://mlsanity.com/paper/2304.04420/) (2023)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [FEAFA+: An Extended Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animation](https://mlsanity.com/paper/2111.02751/) (2021)
- [Unobtrusive Pain Monitoring in Older Adults with Dementia using Pairwise and Contrastive Training](https://mlsanity.com/paper/2101.03251/) (2021)
- [MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection](https://mlsanity.com/paper/2204.01349/) (2022)
- [Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample](https://mlsanity.com/paper/2410.01251/) (2024)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [Safety Case Templates for Autonomous Systems](https://mlsanity.com/paper/2102.02625/) (2021)
- [Self-Supervised Approach for Facial Movement Based Optical Flow](https://mlsanity.com/paper/2105.01256/) (2021)
- [Learning Contrastive Feature Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2402.06165/) (2024)
- [Automatic Facial Paralysis Estimation with Facial Action Units](https://mlsanity.com/paper/2203.01800/) (2022)
- [Attention Based Relation Network for Facial Action Units Recognition](https://mlsanity.com/paper/2210.13988/) (2022)
- [Global-to-local Expression-aware Embeddings for Facial Action Unit Detection](https://mlsanity.com/paper/2210.15160/) (2022)
- [Customizable Avatars with Dynamic Facial Action Coded Expressions (CADyFACE) for Improved User Engagement](https://mlsanity.com/paper/2403.07314/) (2024)

## Abstract (arXiv)

Detecting action units (AUs) on human faces is challenging because various AUs make subtle facial appearance change over various regions at different scales. Current works have attempted to recognize AUs by emphasizing important regions. However, the incorporation of expert prior knowledge into region definition remains under-exploited, and current AU detection approaches do not use regional convolutional neural networks (R-CNN) with expert prior knowledge to directly focus on AU-related regions adaptively. By incorporating expert prior knowledge, we propose a novel R-CNN based model named AU R-CNN. The proposed solution offers two main contributions: (1) AU R-CNN directly observes different facial regions, where various AUs are located. Specifically, we define an AU partition rule which encodes the expert prior knowledge into the region definition and RoI-level label definition. This design produces considerably better detection performance than existing approaches. (2) We integrate various dynamic models (including convolutional long short-term memory, two stream network, conditional random field, and temporal action localization network) into AU R-CNN and then investigate and analyze the reason behind the performance of dynamic models. Experiment results demonstrate that textitonly static RGB image information and no optical flow-based AU R-CNN surpasses the one fused with dynamic models. AU R-CNN is also superior to traditional CNNs that use the same backbone on varying image resolutions. State-of-the-art recognition performance of AU detection is achieved. The complete network is end-to-end trainable. Experiments on BP4D and DISFA datasets show the effectiveness of our approach. The implementation code is available online.