mlsanityData to 7 Oct 2026

ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech

arXiv 2209.07556 ↗First posted 15 Sep 2022cs.GR cs.LG cs.SDcited by 83

Not on the map. Why?

Abstract (arXiv)

We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the same input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

22 of 41 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2015Generating Sentences from a Continuous Spacecited by 607
  6. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  7. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  8. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  10. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  11. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  12. 2019Learning Individual Styles of Conversational Gesturecited by 190
10 more
  1. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  2. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  3. 2018Variational Autoencoders Pursue PCA Directions (by Accident)cited by 84
  4. 2020Gesticulator: A framework for semantically-aware speech-driven gesture generationcited by 84
  5. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  6. 2018Recurrent Transition Networks for Character Locomotioncited by 52
  7. 2021A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020cited by 38
  8. 2021HEMVIP: Human Evaluation of Multiple Videos in Parallelcited by 13
  9. 2021Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speechcited by 11
  10. 2021SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agentscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 23; 2025: 23; 2026: 14.

83 papers.

Sort:

By area:

  1. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  2. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  3. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  4. 2023Human Motion Generation: A Surveycited by 95
  5. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  6. 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
  7. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  8. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  9. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
  10. 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
  11. 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
  12. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
38 more
  1. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 30
  2. 2023Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representationcited by 27
  3. 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
  4. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  5. 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
  6. 2024Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalnesscited by 18
  7. 2023Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generationcited by 17
  8. 2025Embody 3D: A Large-scale Multimodal Motion and Behavior Datasetcited by 16
  9. 2023EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Modelcited by 15
  10. 2023UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletonscited by 15
  11. 2023The DiffuseStyleGesture+ entry to the GENEA Challenge 2023cited by 14
  12. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  13. 2023A Unified Framework for Multimodal, Multi-Part Human Motion Synthesiscited by 12
  14. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 10
  15. 2023Co-Speech Gesture Synthesis using Discrete Gesture Token Learningcited by 9
  16. 2025Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusioncited by 9
  17. 2026MIBURI: Towards Expressive Interactive Gesture Synthesiscited by 9
  18. 2023AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesiscited by 8
  19. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  20. 2024InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarioscited by 7
  21. 2025StickMotion: Generating 3D Human Motions by Drawing a Stickmancited by 7
  22. 2025Model See Model Do: Speech-Driven Facial Animation with Style Controlcited by 7
  23. 2026PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakerscited by 7
  24. 2023GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPTcited by 6
  25. 2023Large language models in textual analysis for gesture selectioncited by 6
  26. 2024Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inferencecited by 6
  27. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  28. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  29. 2024LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesiscited by 5
  30. 2024Scalable Group Choreography via Variational Phase Manifold Learningcited by 4
  31. 2024Decoupling Contact for Fine-Grained Motion Style Transfercited by 4
  32. 2025CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generationcited by 4
  33. 2024Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesiscited by 3
  34. 2025DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speechcited by 3
  35. 2025fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignmentcited by 3
  36. 2025InteracTalker: Prompt-Based Human-Object Interaction with Co-Speech Gesture Generationcited by 3
  37. 2026PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generationcited by 3
  38. 2026RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interactioncited by 3

50 most cited of 83 All 83 (JSON)