# TransFER: Learning Relation-aware Facial Expression Representations with Transformers

arXiv: https://arxiv.org/abs/2108.11116 · HTML: https://mlsanity.com/paper/2108.11116/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [Covariance Pooling For Facial Expression Recognition](https://mlsanity.com/paper/1805.04855/) (2018)
- [Automatic Facial Expression Recognition Using Features of Salient Facial Patches](https://mlsanity.com/paper/1505.04026/) (2015)

## Built on by (most cited first, 30 of 85)

- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition](https://mlsanity.com/paper/2207.10299/) (2022)
- [POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition](https://mlsanity.com/paper/2204.04083/) (2022)
- [Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.10335/) (2022)
- [SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation](https://mlsanity.com/paper/2308.11509/) (2023)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition](https://mlsanity.com/paper/2312.04293/) (2023)
- [Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](https://mlsanity.com/paper/2212.05463/) (2022)
- [Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin](https://mlsanity.com/paper/2203.12341/) (2022)
- [Learning Emotion Representations from Verbal and Nonverbal Communication](https://mlsanity.com/paper/2305.13500/) (2023)
- [CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition](https://mlsanity.com/paper/2303.00193/) (2023)
- [NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2206.04975/) (2022)
- [POSTER++: A simpler and stronger facial expression recognition network](https://mlsanity.com/paper/2301.12149/) (2023)
- [Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training](https://mlsanity.com/paper/2211.10801/) (2022)
- [Towards Evaluating Explanations of Vision Transformers for Medical Imaging](https://mlsanity.com/paper/2304.06133/) (2023)
- [Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition](https://mlsanity.com/paper/2203.13052/) (2022)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [LA-Net: Landmark-Aware Learning for Reliable Facial Expression Recognition under Label Noise](https://mlsanity.com/paper/2307.09023/) (2023)
- [Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition](https://mlsanity.com/paper/2310.19636/) (2023)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [ReSup: Reliable Label Noise Suppression for Facial Expression Recognition](https://mlsanity.com/paper/2305.17895/) (2023)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues](https://mlsanity.com/paper/2402.00281/) (2024)
- [FeatER: An Efficient Network for Human Reconstruction via Feature Map-Based TransformER](https://mlsanity.com/paper/2205.15448/) (2022)
- [Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers](https://mlsanity.com/paper/2207.11081/) (2022)
- [Revisiting Self-Supervised Contrastive Learning for Facial Expression Recognition](https://mlsanity.com/paper/2210.03853/) (2022)
- [Minutiae-Guided Fingerprint Embeddings via Vision Transformers](https://mlsanity.com/paper/2210.13994/) (2022)
- [ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning](https://mlsanity.com/paper/2305.01486/) (2023)
- [AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels](https://mlsanity.com/paper/2410.22506/) (2024)

## Abstract (arXiv)

Facial expression recognition (FER) has received increasing interest in computer vision. We propose the TransFER model which can learn rich relation-aware local representations. It mainly consists of three components: Multi-Attention Dropping (MAD), ViT-FER, and Multi-head Self-Attention Dropping (MSAD). First, local patches play an important role in distinguishing various expressions, however, few existing works can locate discriminative and diverse local patches. This can cause serious problems when some patches are invisible due to pose variations or viewpoint changes. To address this issue, the MAD is proposed to randomly drop an attention map. Consequently, models are pushed to explore diverse local patches adaptively. Second, to build rich relations between different local patches, the Vision Transformers (ViT) are used in FER, called ViT-FER. Since the global scope is used to reinforce each local patch, a better representation is obtained to boost the FER performance. Thirdly, the multi-head self-attention allows ViT to jointly attend to features from different information subspaces at different positions. Given no explicit guidance, however, multiple self-attentions may extract similar relations. To address this, the MSAD is proposed to randomly drop one self-attention module. As a result, models are forced to learn rich relations among diverse local patches. Our proposed TransFER model outperforms the state-of-the-art methods on several FER benchmarks, showing its effectiveness and usefulness.