# Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition

arXiv: https://arxiv.org/abs/2105.05708 · HTML: https://mlsanity.com/paper/2105.05708/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 19 of 19)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [A Riemannian Network for SPD Matrix Learning](https://mlsanity.com/paper/1608.04233/) (2016)
- [Building Deep Networks on Grassmann Manifolds](https://mlsanity.com/paper/1611.05742/) (2016)
- [Deep Learning on Lie Groups for Skeleton-based Action Recognition](https://mlsanity.com/paper/1612.05877/) (2016)
- [Covariance Pooling For Facial Expression Recognition](https://mlsanity.com/paper/1805.04855/) (2018)
- [FPNN: Field Probing Neural Networks for 3D Data](https://mlsanity.com/paper/1605.06240/) (2016)
- [Kernel Methods on the Riemannian Manifold of Symmetric Positive Definite Matrices](https://mlsanity.com/paper/1412.4172/) (2014)
- [Deep manifold-to-manifold transforming network for action recognition](https://mlsanity.com/paper/1705.10732/) (2017)
- [DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-grained Image Recognition](https://mlsanity.com/paper/1711.04047/) (2017)
- [Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition](https://mlsanity.com/paper/1803.05846/) (2018)
- [Multi-Region Ensemble Convolutional Neural Network for Facial Expression Recognition](https://mlsanity.com/paper/1807.10575/) (2018)
- [Expanding the Family of Grassmannian Kernels: An Embedding Perspective](https://mlsanity.com/paper/1407.1123/) (2014)
- [2D+3D Facial Expression Recognition via Discriminative Dynamic Range Enhancement and Multi-Scale Learning](https://mlsanity.com/paper/2011.08333/) (2020)
- [Local Shape Spectrum Analysis for 3D Facial Expression Recognition](https://mlsanity.com/paper/1705.06900/) (2017)
- [Face-GCN: A Graph Convolutional Network for 3D Dynamic Face Identification/Recognition](https://mlsanity.com/paper/2104.09145/) (2021)

## Abstract (arXiv)

Facial expressions recognition (FER) of 3D face scans has received a significant amount of attention in recent years. Most of the facial expression recognition methods have been proposed using mainly 2D images. These methods suffer from several issues like illumination changes and pose variations. Moreover, 2D mapping from 3D images may lack some geometric and topological characteristics of the face. Hence, to overcome this problem, a multi-modal 2D + 3D feature-based method is proposed. We extract shallow features from the 3D images, and deep features using Convolutional Neural Networks (CNN) from the transformed 2D images. Combining these features into a compact representation uses covariance matrices as descriptors for both features instead of single-handedly descriptors. A covariance matrix learning is used as a manifold layer to reduce the deep covariance matrices size and enhance their discrimination power while preserving their manifold structure. We then use the Bag-of-Features (BoF) paradigm to quantize the covariance matrices after flattening. Accordingly, we obtained two codebooks using shallow and deep features. The global codebook is then used to feed an SVM classifier. High classification performances have been achieved on the BU-3DFE and Bosphorus datasets compared to the state-of-the-art methods.