ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis
Not on the map. Why?
Abstract (arXiv)
Audio-driven human gesture synthesis is a crucial task with broad applications in virtual avatars, human-computer interaction, and creative content generation. Despite notable progress, existing methods often produce gestures that are coarse, lack expressiveness, and fail to fully align with audio semantics. To address these challenges, we propose ExGes, a novel retrieval-enhanced diffusion framework with three key designs: (1) a Motion Base Construction, which builds a gesture library using training dataset; (2) a Motion Retrieval Module, employing constrative learning and momentum distillation for fine-grained reference poses retreiving; and (3) a Precision Control Module, integrating partial masking and stochastic masking to enable flexible and fine-grained control. Experimental evaluations on BEAT2 demonstrate that ExGes reduces Fr'echet Gesture Distance by 6.2% and improves motion diversity by 5.3% over EMAGE, with user studies revealing a 71.3% preference for its naturalness and semantic relevance. Code will be released upon acceptance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 9 Mar 2025.
Builds on
46 of 67 references.
References · 46
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2017Neural Discrete Representation Learningcited by 4,650
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
- 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
- 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
- 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
- 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
- 2019Learning Individual Styles of Conversational Gesturecited by 190
34 more
- 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
- 2022Generating Holistic 3D Human Motion from Speechcited by 180
- 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
- 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
- 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
- 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
- 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 116
- 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
- 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 106
- 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
- 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
- 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
- 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
- 2024Flexible Motion In-betweening with Diffusion Modelscited by 69
- 2023A Comprehensive Review of Data-Driven Co-Speech Gesture Generationcited by 66
- 2023Priority-Centric Human Motion Generation in Discrete Latent Spacecited by 66
- 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
- 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 59
- 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
- 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
- 2023Human Pose as Compositional Tokenscited by 45
- 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
- 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
- 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
- 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39
- 2024Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesiscited by 39
- 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
- 2022Reduce Information Loss in Transformers for Pluralistic Image Inpaintingcited by 36
- 2024Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generationcited by 35
- 2024SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasiscited by 32
- 2024Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesiscited by 25
- 2021GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interactioncited by 18
- 2024SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Modelscited by 18
- 2021TrajeVAE: Controllable Human Motion Generation from Trajectoriescited by 14
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.