# Expression Snippet Transformer for Robust Video-based Facial Expression Recognition

arXiv: https://arxiv.org/abs/2109.08409 · HTML: https://mlsanity.com/paper/2109.08409/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 67 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Conformer: Local Features Coupling Global Representations for Visual Recognition](https://mlsanity.com/paper/2105.03889/) (2021)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [Frame attention networks for facial expression recognition in videos](https://mlsanity.com/paper/1907.00193/) (2019)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [Peak-Piloted Deep Network for Facial Expression Recognition](https://mlsanity.com/paper/1607.06997/) (2016)
- [Temporal Multimodal Fusion for Video Emotion Classification in the Wild](https://mlsanity.com/paper/1709.07200/) (2017)
- [Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video](https://mlsanity.com/paper/2010.10637/) (2020)
- [Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video](https://mlsanity.com/paper/1711.04598/) (2017)
- [FERAtt: Facial Expression Recognition with Attention Net](https://mlsanity.com/paper/1902.03284/) (2019)
- [Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition](https://mlsanity.com/paper/2008.02655/) (2020)
- [Deep generative-contrastive networks for facial expression recognition](https://mlsanity.com/paper/1703.07140/) (2017)
- [Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition](https://mlsanity.com/paper/1811.06937/) (2018)
- [Emotion Recognition with Spatial Attention and Temporal Softmax Pooling](https://mlsanity.com/paper/1910.01254/) (2019)

## Built on by (most cited first, 30 of 33)

- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2307.02227/) (2023)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs](https://mlsanity.com/paper/2407.02157/) (2024)
- [Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2310.05804/) (2023)
- [SVFAP: Self-supervised Video Facial Affect Perceiver](https://mlsanity.com/paper/2401.00416/) (2023)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild](https://mlsanity.com/paper/2404.09010/) (2024)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues](https://mlsanity.com/paper/2402.00281/) (2024)
- [Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras](https://mlsanity.com/paper/2304.10211/) (2023)
- [ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning](https://mlsanity.com/paper/2305.01486/) (2023)
- [Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data](https://mlsanity.com/paper/2409.06154/) (2024)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)
- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)
- [FE-Adapter: Adapting Image-based Emotion Classifiers to Videos](https://mlsanity.com/paper/2408.02421/) (2024)
- [Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting](https://mlsanity.com/paper/2404.17100/) (2024)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)
- [PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2503.16945/) (2025)
- [CS3D: An Efficient Facial Expression Recognition via Event Vision](https://mlsanity.com/paper/2512.09592/) (2025)
- [Context-Aware Academic Emotion Dataset and Benchmark](https://mlsanity.com/paper/2507.00586/) (2025)
- [ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognition](https://mlsanity.com/paper/2511.22188/) (2025)
- [Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors](https://mlsanity.com/paper/2512.16485/) (2025)
- [LLDif: Diffusion Models for Low-light Emotion Recognition](https://mlsanity.com/paper/2408.04235/) (2024)
- [GReFEL: Geometry-Aware Reliable Facial Expression Learning under Bias and Imbalanced Data Distribution](https://mlsanity.com/paper/2410.15927/) (2024)
- [Few-Shot Object Detection via Spatial-Channel State Space Model](https://mlsanity.com/paper/2507.15308/) (2025)
- [User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals](https://mlsanity.com/paper/2507.23544/) (2025)
- [Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2511.10958/) (2025)

## Abstract (arXiv)

The recent success of Transformer has provided a new direction to various visual understanding tasks, including video-based facial expression recognition (FER). By modeling visual relations effectively, Transformer has shown its power for describing complicated patterns. However, Transformer still performs unsatisfactorily to notice subtle facial expression movements, because the expression movements of many videos can be too small to extract meaningful spatial-temporal relations and achieve robust performance. To this end, we propose to decompose each video into a series of expression snippets, each of which contains a small number of facial movements, and attempt to augment the Transformer's ability for modeling intra-snippet and inter-snippet visual relations, respectively, obtaining the Expression snippet Transformer (EST). In particular, for intra-snippet modeling, we devise an attention-augmented snippet feature extractor (AA-SFE) to enhance the encoding of subtle facial movements of each snippet by gradually attending to more salient information. In addition, for inter-snippet modeling, we introduce a shuffled snippet order prediction (SSOP) head and a corresponding loss to improve the modeling of subtle motion changes across subsequent snippets by training the Transformer to identify shuffled snippet orders. Extensive experiments on four challenging datasets (i.e., BU-3DFE, MMI, AFEW, and DFEW) demonstrate that our EST is superior to other CNN-based methods, obtaining state-of-the-art performance.