# Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gestures

arXiv: https://arxiv.org/abs/2008.07643 · HTML: https://mlsanity.com/paper/2008.07643/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Audio-driven Neural Gesture Reenactment with Video Motion Graphs](https://mlsanity.com/paper/2207.11524/) (2022)
- [Multimodal analysis of the predictability of hand-gesture properties](https://mlsanity.com/paper/2108.05762/) (2021)
- [Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech](https://mlsanity.com/paper/2106.14736/) (2021)
- [GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT](https://mlsanity.com/paper/2303.13013/) (2023)
- [Leveraging Speech for Gesture Detection in Multimodal Communication](https://mlsanity.com/paper/2404.14952/) (2024)

## Abstract (arXiv)

Communicative gestures and speech acoustic are tightly linked. Our objective is to predict the timing of gestures according to the acoustic. That is, we want to predict when a certain gesture occurs. We develop a model based on a recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech acoustic and the gesture phases and types have been annotated. The input of the model is a sequence of speech acoustic and the output is a sequence of gesture classes. The classes we are using for the model output is based on a combination of gesture phases and gesture types. We use a sequence comparison technique to evaluate the model performance. We find that the model can predict better certain gesture classes than others. We also perform ablation studies which reveal that fundamental frequency is a relevant feature for gesture prediction task. In another sub-experiment, we find that including eyebrow movements as acting as beat gesture improves the performance. Besides, we also find that a model trained on the data of one given speaker also works for the other speaker of the same conversation. We also perform a subjective experiment to measure how respondents judge the naturalness, the time consistency, and the semantic consistency of the generated gesture timing of a virtual agent. Our respondents rate the output of our model favorably.