# Covariance Pooling For Facial Expression Recognition

arXiv: https://arxiv.org/abs/1805.04855 · HTML: https://mlsanity.com/paper/1805.04855/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2018.

## Built on by (most cited first, 30 of 37)

- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG](https://mlsanity.com/paper/2206.01323/) (2022)
- [Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition](https://mlsanity.com/paper/2104.05160/) (2021)
- [Deep Learning for Micro-expression Recognition: A Survey](https://mlsanity.com/paper/2107.02823/) (2021)
- [SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation](https://mlsanity.com/paper/2308.11509/) (2023)
- [Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](https://mlsanity.com/paper/2212.05463/) (2022)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [Faceptor: A Generalist Model for Face Perception](https://mlsanity.com/paper/2403.09500/) (2024)
- [Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition](https://mlsanity.com/paper/2106.03487/) (2021)
- [Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher](https://mlsanity.com/paper/2205.14361/) (2022)
- [SPDIM: Source-Free Unsupervised Conditional and Label Shift Adaptation in EEG](https://mlsanity.com/paper/2411.07249/) (2024)
- [Disentangling Identity and Pose for Facial Expression Recognition](https://mlsanity.com/paper/2208.08106/) (2022)
- [Deep Riemannian Networks for End-to-End EEG Decoding](https://mlsanity.com/paper/2212.10426/) (2022)
- [A Unified and Interpretable Emotion Representation and Expression Generation](https://mlsanity.com/paper/2404.01243/) (2024)
- [SLOSH: Set LOcality Sensitive Hashing via Sliced-Wasserstein Embeddings](https://mlsanity.com/paper/2112.05872/) (2021)
- [Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry](https://mlsanity.com/paper/2407.10484/) (2024)
- [FER-former: Multi-modal Transformer for Facial Expression Recognition](https://mlsanity.com/paper/2303.12997/) (2023)
- [Using Self-Supervised Auxiliary Tasks to Improve Fine-Grained Facial Representation](https://mlsanity.com/paper/2105.06421/) (2021)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network](https://mlsanity.com/paper/2203.14045/) (2022)
- [Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training](https://mlsanity.com/paper/2107.05736/) (2021)
- [Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition](https://mlsanity.com/paper/2107.11061/) (2021)
- [Kernel function impact on convolutional neural networks](https://mlsanity.com/paper/2302.10266/) (2023)
- [Equivariant vs. Invariant Layers: A Comparison of Backbone and Pooling for Point Cloud Classification](https://mlsanity.com/paper/2306.05553/) (2023)
- [BrainRiem: Riemannian Prototype Learning for Source-Free Cross-Site Brain Network Diagnosis](https://mlsanity.com/paper/2606.29200/) (2026)
- [StuArt: Individualized Classroom Observation of Students with Automatic Behavior Recognition and Tracking](https://mlsanity.com/paper/2211.03127/) (2022)
- [Linear optimal transport subspaces for point set classification](https://mlsanity.com/paper/2403.10015/) (2024)
- [Multi-threshold Deep Metric Learning for Facial Expression Recognition](https://mlsanity.com/paper/2406.16434/) (2024)
- [Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation](https://mlsanity.com/paper/2603.19961/) (2026)

## Abstract (arXiv)

Classifying facial expressions into different categories requires capturing regional distortions of facial landmarks. We believe that second-order statistics such as covariance is better able to capture such distortions in regional facial fea- tures. In this work, we explore the benefits of using a man- ifold network structure for covariance pooling to improve facial expression recognition. In particular, we first employ such kind of manifold networks in conjunction with tradi- tional convolutional networks for spatial pooling within in- dividual image feature maps in an end-to-end deep learning manner. By doing so, we are able to achieve a recognition accuracy of 58.14% on the validation set of Static Facial Expressions in the Wild (SFEW 2.0) and 87.0% on the vali- dation set of Real-World Affective Faces (RAF) Database. Both of these results are the best results we are aware of. Besides, we leverage covariance pooling to capture the tem- poral evolution of per-frame features for video-based facial expression recognition. Our reported results demonstrate the advantage of pooling image-set features temporally by stacking the designed manifold network of covariance pool-ing on top of convolutional network layers.