# Identity-aware Facial Expression Recognition in Compressed Video

arXiv: https://arxiv.org/abs/2101.00317 · HTML: https://mlsanity.com/paper/2101.00317/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 71 references that are in the map.
- First posted on arXiv 1 Jan 2021.

## Builds on (most cited first, 22 of 22)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [Frame attention networks for facial expression recognition in videos](https://mlsanity.com/paper/1907.00193/) (2019)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)
- [Importance-Aware Semantic Segmentation in Self-Driving with Discrete Wasserstein Training](https://mlsanity.com/paper/2010.12440/) (2020)
- [Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models](https://mlsanity.com/paper/1911.07421/) (2019)
- [EmoNets: Multimodal deep learning approaches for emotion recognition in video](https://mlsanity.com/paper/1503.01800/) (2015)
- [Temporal Multimodal Fusion for Video Emotion Classification in the Wild](https://mlsanity.com/paper/1709.07200/) (2017)
- [AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation](https://mlsanity.com/paper/2007.06620/) (2020)
- [Unimodal-uniform Constrained Wasserstein Training for Medical Diagnosis](https://mlsanity.com/paper/1911.02475/) (2019)
- [Dependency-aware Attention Control for Unconstrained Face Recognition with Image Sets](https://mlsanity.com/paper/1907.03030/) (2019)
- [Conservative Wasserstein Training for Pose Estimation](https://mlsanity.com/paper/1911.00962/) (2019)
- [Permutation-invariant Feature Restructuring for Correlation-aware Image Set-based Recognition](https://mlsanity.com/paper/1908.01174/) (2019)
- [Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition](https://mlsanity.com/paper/1811.06937/) (2018)
- [LBVCNN: Local Binary Volume Convolutional Neural Network for Facial Expression Recognition from Image Sequences](https://mlsanity.com/paper/1904.07647/) (2019)
- [Attention Control with Metric Learning Alignment for Image Set-based Recognition](https://mlsanity.com/paper/1908.01872/) (2019)
- [All-In-One: Facial Expression Transfer, Editing and Recognition Using A Single Network](https://mlsanity.com/paper/1911.07050/) (2019)
- [LOMo: Latent Ordinal Model for Facial Analysis in Videos](https://mlsanity.com/paper/1604.01500/) (2016)

## Built on by (most cited first, 9 of 9)

- [Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives](https://mlsanity.com/paper/2208.07422/) (2022)
- [Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate](https://mlsanity.com/paper/2107.13469/) (2021)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues](https://mlsanity.com/paper/2402.00281/) (2024)
- [Recursively Conditional Gaussian for Ordinal Unsupervised Domain Adaptation](https://mlsanity.com/paper/2107.13467/) (2021)
- [Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition](https://mlsanity.com/paper/2204.11053/) (2022)
- [Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation](https://mlsanity.com/paper/2106.13393/) (2021)
- [Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions](https://mlsanity.com/paper/2604.11730/) (2026)
- [Deep 3D-CNN for Depression Diagnosis with Facial Video Recording of Self-Rating Depression Scale Questionnaire](https://mlsanity.com/paper/2107.10712/) (2021)

## Abstract (arXiv)

This paper targets to explore the inter-subject variations eliminated facial expression representation in the compressed video domain. Most of the previous methods process the RGB images of a sequence, while the off-the-shelf and valuable expression-related muscle movement already embedded in the compression format. In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames and possible to extract identity factors from the I frame with a pre-trained face recognition network. By enforcing the marginal independent of them, the expression feature is expected to be purer for the expression and be robust to identity shifts. We do not need the identity label or multiple expression samples from the same person for identity elimination. Moreover, when the apex frame is annotated in the dataset, the complementary constraint can be further added to regularize the feature-level game. In testing, only the compressed residual frames are required to achieve expression prediction. Our solution can achieve comparable or better performance than the recent decoded image based methods on the typical FER benchmarks with about 3× faster inference with compressed data.