# Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction

arXiv: https://arxiv.org/abs/1901.01805 · HTML: https://mlsanity.com/paper/1901.01805/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2019.

## Built on by (most cited first, 8 of 8)

- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild](https://mlsanity.com/paper/2105.07484/) (2021)
- [Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition](https://mlsanity.com/paper/2106.03487/) (2021)
- [Using Scene and Semantic Features for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2308.00228/) (2023)
- [PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2312.12334/) (2023)
- [Evaluating Temporal Patterns in Applied Infant Affect Recognition](https://mlsanity.com/paper/2209.03496/) (2022)
- [Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective](https://mlsanity.com/paper/2302.10021/) (2023)
- [MoEmo Vision Transformer: Integrating Cross-Attention and Movement Vectors in 3D Pose Estimation for HRI Emotion Detection](https://mlsanity.com/paper/2310.09757/) (2023)

## Abstract (arXiv)

In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child-robot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.