# Towards Inclusive HRI: Using Sim2Real to Address Underrepresentation in Emotion Expression Recognition

arXiv: https://arxiv.org/abs/2208.07472 · HTML: https://mlsanity.com/paper/2208.07472/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 15 Aug 2022.

## Builds on (most cited first, 14 of 14)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB](https://mlsanity.com/paper/1712.01057/) (2017)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [A Deeper Look at Dataset Bias](https://mlsanity.com/paper/1505.01257/) (2015)
- [Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems](https://mlsanity.com/paper/1712.01619/) (2017)
- [Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions](https://mlsanity.com/paper/1708.00370/) (2017)
- [Improving Face Detection Performance with 3D-Rendered Synthetic Data](https://mlsanity.com/paper/1812.07363/) (2018)
- [Developing a Data-Driven Categorical Taxonomy of Emotional Expressions in Real World Human Robot Interactions](https://mlsanity.com/paper/2103.04262/) (2021)

## Built on by (most cited first, 2 of 2)

- [ImaginaryNet: Learning Object Detectors without Real Images and Annotations](https://mlsanity.com/paper/2210.06886/) (2022)
- [Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion](https://mlsanity.com/paper/2302.06683/) (2023)

## Abstract (arXiv)

Robots and artificial agents that interact with humans should be able to do so without bias and inequity, but facial perception systems have notoriously been found to work more poorly for certain groups of people than others. In our work, we aim to build a system that can perceive humans in a more transparent and inclusive manner. Specifically, we focus on dynamic expressions on the human face, which are difficult to collect for a broad set of people due to privacy concerns and the fact that faces are inherently identifiable. Furthermore, datasets collected from the Internet are not necessarily representative of the general population. We address this problem by offering a Sim2Real approach in which we use a suite of 3D simulated human models that enables us to create an auditable synthetic dataset covering 1) underrepresented facial expressions, outside of the six basic emotions, such as confusion; 2) ethnic or gender minority groups; and 3) a wide range of viewing angles that a robot may encounter a human in the real world. By augmenting a small dynamic emotional expression dataset containing 123 samples with a synthetic dataset containing 4536 samples, we achieved an improvement in accuracy of 15% on our own dataset and 11% on an external benchmark dataset, compared to the performance of the same model architecture without synthetic training data. We also show that this additional step improves accuracy specifically for racial minorities when the architecture's feature extraction weights are trained from scratch.