# CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition

arXiv: https://arxiv.org/abs/2303.00193 · HTML: https://mlsanity.com/paper/2303.00193/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion](https://mlsanity.com/paper/2103.16854/) (2021)
- [Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.10335/) (2022)

## Built on by (most cited first, 28 of 28)

- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition](https://mlsanity.com/paper/2310.16640/) (2023)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts](https://mlsanity.com/paper/2308.01313/) (2023)
- [VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning](https://mlsanity.com/paper/2404.07078/) (2024)
- [ExpLLM: Towards Chain of Thought for Facial Expression Recognition](https://mlsanity.com/paper/2409.02828/) (2024)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild](https://mlsanity.com/paper/2404.09010/) (2024)
- [Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data](https://mlsanity.com/paper/2409.06154/) (2024)
- [Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2603.21309/) (2026)
- [OUS: Scene-Guided Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2405.18769/) (2024)
- [Multimodal Prompt Alignment for Facial Expression Recognition](https://mlsanity.com/paper/2506.21017/) (2025)
- [D2SP: Dynamic Dual-Stage Purification Framework for Dual Noise Mitigation in Vision-based Affective Recognition](https://mlsanity.com/paper/2406.16473/) (2024)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)
- [Hierarchical Vision-Language Interaction for Facial Action Unit Detection](https://mlsanity.com/paper/2602.14425/) (2026)
- [Hi-EF: Benchmarking Emotion Forecasting in Human-interaction](https://mlsanity.com/paper/2407.16406/) (2024)
- [Context-Aware Academic Emotion Dataset and Benchmark](https://mlsanity.com/paper/2507.00586/) (2025)
- [Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description](https://mlsanity.com/paper/2410.02049/) (2024)
- [Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2502.21004/) (2025)
- [From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition](https://mlsanity.com/paper/2507.11892/) (2025)
- [Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2511.10958/) (2025)
- [Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets](https://mlsanity.com/paper/2604.10541/) (2026)
- [Cognition-Inspired Dual-Stream Semantic Enhancement for Vision-Based Dynamic Emotion Modeling](https://mlsanity.com/paper/2604.12777/) (2026)
- [Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2604.25255/) (2026)
- [LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition](https://mlsanity.com/paper/2605.19821/) (2026)
- [Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding](https://mlsanity.com/paper/2606.30611/) (2026)
- [A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources](https://mlsanity.com/paper/2609.13255/) (2026)

## Abstract (arXiv)

Facial expression recognition (FER) is an essential task for understanding human behaviors. As one of the most informative behaviors of humans, facial expressions are often compound and variable, which is manifested by the fact that different people may express the same expression in very different ways. However, most FER methods still use one-hot or soft labels as the supervision, which lack sufficient semantic descriptions of facial expressions and are less interpretable. Recently, contrastive vision-language pre-training (VLP) models (e.g., CLIP) use text as supervision and have injected new vitality into various computer vision tasks, benefiting from the rich semantics in text. Therefore, in this work, we propose CLIPER, a unified framework for both static and dynamic facial Expression Recognition based on CLIP. Besides, we introduce multiple expression text descriptors (METD) to learn fine-grained expression representations that make CLIPER more interpretable. We conduct extensive experiments on several popular FER benchmarks and achieve state-of-the-art performance, which demonstrates the effectiveness of CLIPER.