# Transformer Network for Semantically-Aware and Speech-Driven Upper-Face Generation

arXiv: https://arxiv.org/abs/2110.04527 · HTML: https://mlsanity.com/paper/2110.04527/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2021.

## Built on by (most cited first, 3 of 3)

- [AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2305.01241/) (2023)
- [Large language models in textual analysis for gesture selection](https://mlsanity.com/paper/2310.13705/) (2023)
- [META4: Semantically-Aligned Generation of Metaphoric Gestures Using Self-Supervised Text and Speech Representation](https://mlsanity.com/paper/2311.05481/) (2023)

## Abstract (arXiv)

We propose a semantically-aware speech driven model to generate expressive and natural upper-facial and head motion for Embodied Conversational Agents (ECA). In this work, we aim to produce natural and continuous head motion and upper-facial gestures synchronized with speech. We propose a model that generates these gestures based on multimodal input features: the first modality is text, and the second one is speech prosody. Our model makes use of Transformers and Convolutions to map the multimodal features that correspond to an utterance to continuous eyebrows and head gestures. We conduct subjective and objective evaluations to validate our approach and compare it with state of the art.