# FE-Adapter: Adapting Image-based Emotion Classifiers to Videos

arXiv: https://arxiv.org/abs/2408.02421 · HTML: https://mlsanity.com/paper/2408.02421/

## Key facts

- Cited by 4 papers in the map (data through 7 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [Space-time Mixing Attention for Video Transformer](https://mlsanity.com/paper/2106.05968/) (2021)
- [SMART Frame Selection for Action Recognition](https://mlsanity.com/paper/2012.10671/) (2020)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion](https://mlsanity.com/paper/2103.16854/) (2021)
- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2307.02227/) (2023)
- [Intensity-Aware Loss for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.10335/) (2022)
- [Expression Snippet Transformer for Robust Video-based Facial Expression Recognition](https://mlsanity.com/paper/2109.08409/) (2021)
- [ColorNet: Investigating the importance of color spaces for image classification](https://mlsanity.com/paper/1902.00267/) (2019)
- [Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition](https://mlsanity.com/paper/2206.04790/) (2022)
- [D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2111.07774/) (2021)
- [Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition](https://mlsanity.com/paper/2107.03107/) (2021)
- [Watt For What: Rethinking Deep Learning's Energy-Performance Relationship](https://mlsanity.com/paper/2310.06522/) (2023)
- [Vision Transformer Equipped with Neural Resizer on Facial Expression Recognition Task](https://mlsanity.com/paper/2204.02181/) (2022)
- [Optimizing ViViT Training: Time and Memory Reduction for Action Recognition](https://mlsanity.com/paper/2306.04822/) (2023)

## Built on by (most cited first, 4 of 4)

- [PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2503.16945/) (2025)
- [Action Unit Enhance Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2507.07678/) (2025)
- [Is Temporal Prompting All We Need For Limited Labeled Action Recognition?](https://mlsanity.com/paper/2504.01890/) (2025)
- [Enhancing Ambiguous Dynamic Facial Expression Recognition with Soft Label-based Data Augmentation](https://mlsanity.com/paper/2506.20867/) (2025)

## Abstract (arXiv)

Utilizing large pre-trained models for specific tasks has yielded impressive results. However, fully fine-tuning these increasingly large models is becoming prohibitively resource-intensive. This has led to a focus on more parameter-efficient transfer learning, primarily within the same modality. But this approach has limitations, particularly in video understanding where suitable pre-trained models are less common. Addressing this, our study introduces a novel cross-modality transfer learning approach from images to videos, which we call parameter-efficient image-to-video transfer learning. We present the Facial-Emotion Adapter (FE-Adapter), designed for efficient fine-tuning in video tasks. This adapter allows pre-trained image models, which traditionally lack temporal processing capabilities, to analyze dynamic video content efficiently. Notably, it uses about 15 times fewer parameters than previous methods, while improving accuracy. Our experiments in video emotion recognition demonstrate that the FE-Adapter can match or even surpass existing fine-tuning and video emotion models in both performance and efficiency. This breakthrough highlights the potential for cross-modality approaches in enhancing the capabilities of AI models, particularly in fields like video emotion analysis where the demand for efficiency and accuracy is constantly rising.