# Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition

arXiv: https://arxiv.org/abs/2212.05463 · HTML: https://mlsanity.com/paper/2212.05463/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 96 references that are in the map.
- First posted on arXiv 11 Dec 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](https://mlsanity.com/paper/2104.10858/) (2021)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)

## Built on by (most cited first, 30 of 35)

- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues](https://mlsanity.com/paper/2402.00281/) (2024)
- [Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning](https://mlsanity.com/paper/2504.07198/) (2025)
- [ResEmoteNet: Bridging Accuracy and Loss Reduction in Facial Emotion Recognition](https://mlsanity.com/paper/2409.10545/) (2024)
- [AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels](https://mlsanity.com/paper/2410.22506/) (2024)
- [PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition](https://mlsanity.com/paper/2306.09626/) (2023)
- [SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data](https://mlsanity.com/paper/2410.09865/) (2024)
- [Task-adaptive Q-Face](https://mlsanity.com/paper/2405.09059/) (2024)
- [Multimodal Prompt Alignment for Facial Expression Recognition](https://mlsanity.com/paper/2506.21017/) (2025)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)
- [Learning in Focus: Detecting Behavioral and Collaborative Engagement Using Vision Transformers](https://mlsanity.com/paper/2508.15782/) (2025)
- [Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for AH Detection](https://mlsanity.com/paper/2603.13406/) (2026)
- [Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions](https://mlsanity.com/paper/2604.11730/) (2026)
- [Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis](https://mlsanity.com/paper/2303.10511/) (2023)
- [M2Former: Multi-Scale Patch Selection for Fine-Grained Visual Recognition](https://mlsanity.com/paper/2308.02161/) (2023)
- [Batch Transformer: Look for Attention in Batch](https://mlsanity.com/paper/2407.04218/) (2024)
- [Spatial Action Unit Cues for Interpretable Deep Facial Expression Recognition](https://mlsanity.com/paper/2410.01848/) (2024)
- [Explainable AI in Speaker Recognition -- Attention Map Visualisation and Evaluation](https://mlsanity.com/paper/2606.22901/) (2026)
- [Bridging the Gaps: Utilizing Unlabeled Face Recognition Datasets to Boost Semi-Supervised Facial Expression Recognition](https://mlsanity.com/paper/2410.17622/) (2024)
- [ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognition](https://mlsanity.com/paper/2511.22188/) (2025)
- [A Perception CNN for Facial Expression Recognition](https://mlsanity.com/paper/2512.06422/) (2025)
- [SimFLE: Simple Facial Landmark Encoding for Self-Supervised Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2303.07648/) (2023)
- [Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets](https://mlsanity.com/paper/2308.07871/) (2023)
- [Exploiting Diverse Feature for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2308.13421/) (2023)
- [FERT: Real-Time Facial Expression Recognition with Short-Range FMCW Radar](https://mlsanity.com/paper/2411.11619/) (2024)
- [A Visual Self-attention Mechanism Facial Expression Recognition Network beyond Convnext](https://mlsanity.com/paper/2504.09077/) (2025)
- [TKFNet: Learning Texture Key Factor Driven Feature for Facial Expression Recognition](https://mlsanity.com/paper/2505.09967/) (2025)
- [Semi-Supervised Facial Expression Recognition based on Dynamic Threshold and Negative Learning](https://mlsanity.com/paper/2601.05556/) (2026)
- [Emotion Diffusion Classifier with Adaptive Margin Discrepancy Training for Facial Expression Recognition](https://mlsanity.com/paper/2603.29578/) (2026)

## Abstract (arXiv)

Facial Expression Recognition (FER) in the wild is an extremely challenging task. Recently, some Vision Transformers (ViT) have been explored for FER, but most of them perform inferiorly compared to Convolutional Neural Networks (CNN). This is mainly because the new proposed modules are difficult to converge well from scratch due to lacking inductive bias and easy to focus on the occlusion and noisy areas. TransFER, a representative transformer-based method for FER, alleviates this with multi-branch attention dropping but brings excessive computations. On the contrary, we present two attentive pooling (AP) modules to pool noisy features directly. The AP modules include Attentive Patch Pooling (APP) and Attentive Token Pooling (ATP). They aim to guide the model to emphasize the most discriminative features while reducing the impacts of less relevant features. The proposed APP is employed to select the most informative patches on CNN features, and ATP discards unimportant tokens in ViT. Being simple to implement and without learnable parameters, the APP and ATP intuitively reduce the computational cost while boosting the performance by ONLY pursuing the most discriminative features. Qualitative results demonstrate the motivations and effectiveness of our attentive poolings. Besides, quantitative results on six in-the-wild datasets outperform other state-of-the-art methods.