# ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech

arXiv: https://arxiv.org/abs/2209.07556 · HTML: https://mlsanity.com/paper/2209.07556/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Unpaired Motion Style Transfer from Video to Animation](https://mlsanity.com/paper/2005.05751/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Variational Autoencoders Pursue PCA Directions (by Accident)](https://mlsanity.com/paper/1812.06775/) (2018)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach](https://mlsanity.com/paper/2007.12553/) (2020)
- [Recurrent Transition Networks for Character Locomotion](https://mlsanity.com/paper/1810.02363/) (2018)
- [A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020](https://mlsanity.com/paper/2102.11617/) (2021)
- [HEMVIP: Human Evaluation of Multiple Videos in Parallel](https://mlsanity.com/paper/2101.11898/) (2021)
- [Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech](https://mlsanity.com/paper/2107.00712/) (2021)
- [SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents](https://mlsanity.com/paper/2108.04636/) (2021)

## Built on by (most cited first, 30 of 84)

- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models](https://mlsanity.com/paper/2305.04919/) (2023)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2403.17936/) (2024)
- [Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2405.09814/) (2024)
- [Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion](https://mlsanity.com/paper/2312.04466/) (2023)
- [Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model](https://mlsanity.com/paper/2404.01862/) (2024)
- [Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation](https://mlsanity.com/paper/2410.00464/) (2024)
- [Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfer](https://mlsanity.com/paper/2406.06508/) (2024)
- [Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation](https://mlsanity.com/paper/2309.05455/) (2023)
- [Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis](https://mlsanity.com/paper/2412.06786/) (2024)
- [BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer](https://mlsanity.com/paper/2310.06851/) (2023)
- [MotionScript: Natural Language Descriptions for Expressive 3D Human Motions](https://mlsanity.com/paper/2312.12634/) (2023)
- [Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset](https://mlsanity.com/paper/2510.16258/) (2025)
- [Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness](https://mlsanity.com/paper/2401.03476/) (2024)
- [Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation](https://mlsanity.com/paper/2311.17532/) (2023)
- [EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model](https://mlsanity.com/paper/2306.11496/) (2023)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [The DiffuseStyleGesture+ entry to the GENEA Challenge 2023](https://mlsanity.com/paper/2308.13879/) (2023)
- [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023)
- [A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis](https://mlsanity.com/paper/2311.16471/) (2023)
- [Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models](https://mlsanity.com/paper/2507.20220/) (2025)
- [Co-Speech Gesture Synthesis using Discrete Gesture Token Learning](https://mlsanity.com/paper/2303.12822/) (2023)
- [StickMotion: Generating 3D Human Motions by Drawing a Stickman](https://mlsanity.com/paper/2503.04829/) (2025)
- [Co3Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion](https://mlsanity.com/paper/2505.01746/) (2025)
- [MIBURI: Towards Expressive Interactive Gesture Synthesis](https://mlsanity.com/paper/2603.03282/) (2026)

## Abstract (arXiv)

We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the same input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles.