# CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing

arXiv: https://arxiv.org/abs/2101.03503 · HTML: https://mlsanity.com/paper/2101.03503/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 65 references that are in the map.
- First posted on arXiv 10 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates](https://mlsanity.com/paper/1706.08276/) (2017)
- [Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications](https://mlsanity.com/paper/1606.03237/) (2016)
- [Level Playing Field for Million Scale Face Recognition](https://mlsanity.com/paper/1705.00393/) (2017)
- [Speech Emotion Recognition with Dual-Sequence LSTM Architecture](https://mlsanity.com/paper/1910.08874/) (2019)
- [When Face Recognition Meets with Deep Learning: an Evaluation of Convolutional Neural Networks for Face Recognition](https://mlsanity.com/paper/1504.02351/) (2015)
- [Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues](https://mlsanity.com/paper/1501.05703/) (2015)
- [Face Recognition: A Novel Multi-Level Taxonomy based Survey](https://mlsanity.com/paper/1901.00713/) (2019)
- [A Double-Deep Spatio-Angular Learning Framework for Light Field based Face Recognition](https://mlsanity.com/paper/1805.10078/) (2018)

## Built on by (most cited first, 9 of 9)

- [Spatiotemporal Contrastive Learning of Facial Expressions in Videos](https://mlsanity.com/paper/2108.03064/) (2021)
- [Distilling EEG Representations via Capsules for Affective Computing](https://mlsanity.com/paper/2105.00104/) (2021)
- [Face Trees for Expression Recognition](https://mlsanity.com/paper/2112.02487/) (2021)
- [Analysis of Semi-Supervised Methods for Facial Expression Recognition](https://mlsanity.com/paper/2208.00544/) (2022)
- [FaceTopoNet: Facial Expression Recognition using Face Topology Learning](https://mlsanity.com/paper/2209.06322/) (2022)
- [Privacy-Preserving Remote Heart Rate Estimation from Facial Videos](https://mlsanity.com/paper/2306.01141/) (2023)
- [Headset: Human emotion awareness under partial occlusions multimodal dataset](https://mlsanity.com/paper/2402.09107/) (2024)
- [Multi-Perspective LSTM for Joint Visual Representation Learning](https://mlsanity.com/paper/2105.02802/) (2021)
- [Contrastive Learning of View-Invariant Representations for Facial Expressions Recognition](https://mlsanity.com/paper/2311.06852/) (2023)

## Abstract (arXiv)

Light field (LF) cameras provide rich spatio-angular visual representations by sensing the visual scene from multiple perspectives and have recently emerged as a promising technology to boost the performance of human-machine systems such as biometrics and affective computing. Despite the significant success of LF representation for constrained facial image analysis, this technology has never been used for face and expression recognition in the wild. In this context, this paper proposes a new deep face and expression recognition solution, called CapsField, based on a convolutional neural network and an additional capsule network that utilizes dynamic routing to learn hierarchical relations between capsules. CapsField extracts the spatial features from facial images and learns the angular part-whole relations for a selected set of 2D sub-aperture images rendered from each LF image. To analyze the performance of the proposed solution in the wild, the first in the wild LF face dataset, along with a new complementary constrained face dataset captured from the same subjects recorded earlier have been captured and are made available. A subset of the in the wild dataset contains facial images with different expressions, annotated for usage in the context of face expression recognition tests. An extensive performance assessment study using the new datasets has been conducted for the proposed and relevant prior solutions, showing that the CapsField proposed solution achieves superior performance for both face and expression recognition tasks when compared to the state-of-the-art.