mlsanityData to 14 Oct 2026

Real-time Gesture Animation Generation from Speech for Virtual Human Interaction

arXiv 2208.03244 ↗First posted 5 Aug 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose a real-time system for synthesizing gestures directly from speech. Our data-driven approach is based on Generative Adversarial Neural Networks to model the speech-gesture relationship. We utilize the large amount of speaker video data available online to train our 3D gesture model. Our model generates speaker-specific gestures by taking consecutive audio input chunks of two seconds in length. We animate the predicted gestures on a virtual avatar. We achieve a delay below three seconds between the time of audio input and gesture animation. Code and videos are available at https://github.com/mrebol/Gestures-From-Speech

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

7 of 9 references.

References · 7

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  3. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 403
  4. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  5. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  6. 2019Learning Individual Styles of Conversational Gesturecited by 191
  7. 2021Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speechcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
  2. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applicationscited by 5
  3. 2025MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantizationcited by 1
  4. 2023Mixed Reality Communication for Medical Procedures: Teaching the Placement of a Central Venous Cathetercited by 0