# Face-Cap: Image Captioning using Facial Expression Analysis

arXiv: https://arxiv.org/abs/1807.02250 · HTML: https://mlsanity.com/paper/1807.02250/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 3 of 3)

- [Face2Text revisited: Improved data set and baseline results](https://mlsanity.com/paper/2205.12342/) (2022)
- [SafeAccess+: An Intelligent System to make Smart Home Safer and Americans with Disability Act Compliant](https://mlsanity.com/paper/2110.09273/) (2021)
- [FaceGemma: Enhancing Image Captioning with Facial Attributes for Portrait Images](https://mlsanity.com/paper/2309.13601/) (2023)

## Abstract (arXiv)

Image captioning is the process of generating a natural language description of an image. Most current image captioning models, however, do not take into account the emotional aspect of an image, which is very relevant to activities and interpersonal relationships represented therein. Towards developing a model that can produce human-like captions incorporating these, we use facial expression features extracted from images including human faces, with the aim of improving the descriptive ability of the model. In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions. Using all standard evaluation metrics, our Face-Cap models outperform a state-of-the-art baseline model for generating image captions when applied to an image caption dataset extracted from the standard Flickr 30K dataset, consisting of around 11K images containing faces. An analysis of the captions finds that, perhaps surprisingly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions.