# ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis

arXiv: https://arxiv.org/abs/2503.06499 · HTML: https://mlsanity.com/paper/2503.06499/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism](https://mlsanity.com/paper/2309.00796/) (2023)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models](https://mlsanity.com/paper/2305.04919/) (2023)
- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Flexible Motion In-betweening with Diffusion Models](https://mlsanity.com/paper/2405.11126/) (2024)
- [Priority-Centric Human Motion Generation in Discrete Latent Space](https://mlsanity.com/paper/2308.14480/) (2023)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation](https://mlsanity.com/paper/2401.04747/) (2024)
- [Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives](https://mlsanity.com/paper/2403.10518/) (2024)

## Built on by (most cited first, 2 of 2)

- [ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars](https://mlsanity.com/paper/2512.19546/) (2025)
- [SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation](https://mlsanity.com/paper/2609.33311/) (2026)

## Abstract (arXiv)

Audio-driven human gesture synthesis is a crucial task with broad applications in virtual avatars, human-computer interaction, and creative content generation. Despite notable progress, existing methods often produce gestures that are coarse, lack expressiveness, and fail to fully align with audio semantics. To address these challenges, we propose ExGes, a novel retrieval-enhanced diffusion framework with three key designs: (1) a Motion Base Construction, which builds a gesture library using training dataset; (2) a Motion Retrieval Module, employing constrative learning and momentum distillation for fine-grained reference poses retreiving; and (3) a Precision Control Module, integrating partial masking and stochastic masking to enable flexible and fine-grained control. Experimental evaluations on BEAT2 demonstrate that ExGes reduces Fr'echet Gesture Distance by 6.2% and improves motion diversity by 5.3% over EMAGE, with user studies revealing a 71.3% preference for its naturalness and semantic relevance. Code will be released upon acceptance.