mlsanityData to 7 Oct 2026

Multimodal analysis of the predictability of hand-gesture properties

arXiv 2108.05762 ↗First posted 12 Aug 2021cs.HC cs.LG cs.MMcited by 14

Not on the map. Why?

Abstract (arXiv)

Embodied conversational agents benefit from being able to accompany their speech with gestures. Although many data-driven approaches to gesture generation have been proposed in recent years, it is still unclear whether such systems can consistently generate gestures that convey meaning. We investigate which gesture properties (phase, category, and semantics) can be predicted from speech text and/or audio using contemporary deep learning. In extensive experiments, we show that gesture properties related to gesture meaning (semantics and category) are predictable from text features (time-aligned FastText embeddings) alone, but not from prosodic audio features, while rhythm-related gesture properties (phase) on the other hand can be predicted from audio features better than from text. These results are encouraging as they indicate that it is possible to equip an embodied agent with content-wise meaningful co-speech gestures using a machine-learning model.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 17 of its 60 references that are in the map. First posted on arXiv 12 Aug 2021.

Builds on

17 of 60 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  5. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  6. 2016FastText.zip: Compressing text classification modelscited by 331
  7. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  8. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  9. 2018Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robotscited by 110
  10. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  11. 2019Macro F1 and Macro F1cited by 70
  12. 2019Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interactioncited by 48
5 more
  1. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  2. 2020Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generationcited by 20
  3. 2021Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speechcited by 12
  4. 2020Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gesturescited by 5
  5. 2020Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importancecited by 5

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 6; 2024: 5.

14 papers.

Sort:

By area:

  1. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  2. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  3. 2022The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generationcited by 40
  4. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  5. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  6. 2021Integrated Speech and Gesture Synthesiscited by 16
  7. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  8. 2023GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPTcited by 6
  9. 2023Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agentcited by 5
  10. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  11. 2024NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interactioncited by 4
  12. 2024Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluationcited by 3
2 more
  1. 2024Leveraging Speech for Gesture Detection in Multimodal Communicationcited by 2
  2. 2024Large Body Language Modelscited by 0