# Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

arXiv: https://arxiv.org/abs/2312.15900 · HTML: https://mlsanity.com/paper/2312.15900/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation](https://mlsanity.com/paper/2212.03741/) (2022)
- [DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models](https://mlsanity.com/paper/2305.04919/) (2023)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors](https://mlsanity.com/paper/2309.17261/) (2023)
- [Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation](https://mlsanity.com/paper/2307.11545/) (2023)
- [Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates](https://mlsanity.com/paper/2108.08020/) (2021)
- [Audio-Driven Co-Speech Gesture Video Generation](https://mlsanity.com/paper/2212.02350/) (2022)
- [QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation](https://mlsanity.com/paper/2305.11094/) (2023)
- [EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation](https://mlsanity.com/paper/2305.18891/) (2023)
- [EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model](https://mlsanity.com/paper/2306.11496/) (2023)
- [The DiffuseStyleGesture+ entry to the GENEA Challenge 2023](https://mlsanity.com/paper/2308.13879/) (2023)
- [The ReprGesture entry to the GENEA Challenge 2022](https://mlsanity.com/paper/2208.12133/) (2022)

## Built on by (most cited first, 18 of 18)

- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives](https://mlsanity.com/paper/2403.10518/) (2024)
- [MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models](https://mlsanity.com/paper/2403.09471/) (2024)
- [GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling](https://mlsanity.com/paper/2501.18898/) (2025)
- [DisPose: Disentangling Pose Guidance for Controllable Human Image Animation](https://mlsanity.com/paper/2412.09349/) (2024)
- [SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis](https://mlsanity.com/paper/2412.16563/) (2024)
- [Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness](https://mlsanity.com/paper/2401.03476/) (2024)
- [MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation](https://mlsanity.com/paper/2503.19383/) (2025)
- [Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representation](https://mlsanity.com/paper/2502.07239/) (2025)
- [Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion](https://mlsanity.com/paper/2504.00430/) (2025)
- [Alignment is All You Need: A Training-free Augmentation Strategy for Pose-guided Video Generation](https://mlsanity.com/paper/2408.16506/) (2024)
- [Intentional Gesture: Deliver Your Intentions with Gestures for Speech](https://mlsanity.com/paper/2505.15197/) (2025)
- [LiveGesture Streamable Co-Speech Gesture Generation Model](https://mlsanity.com/paper/2604.10927/) (2026)
- [Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation](https://mlsanity.com/paper/2409.10535/) (2024)
- [Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis](https://mlsanity.com/paper/2504.09885/) (2025)
- [CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation](https://mlsanity.com/paper/2511.22863/) (2025)
- [SteerFace: Debiasing Synthetic Face Generation via Adaptive Residue Perturbation](https://mlsanity.com/paper/2605.30894/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)

## Abstract (arXiv)

This study aims to improve the generation of 3D gestures by utilizing multimodal information from human speech. Previous studies have focused on incorporating additional modalities to enhance the quality of generated gestures. However, these methods perform poorly when certain modalities are missing during inference. To address this problem, we suggest using speech-derived multimodal priors to improve gesture generation. We introduce a novel method that separates priors from speech and employs multimodal priors as constraints for generating gestures. Our approach utilizes a chain-like modeling method to generate facial blendshapes, body movements, and hand gestures sequentially. Specifically, we incorporate rhythm cues derived from facial deformation and stylization prior based on speech emotions, into the process of generating gestures. By incorporating multimodal priors, our method improves the quality of generated gestures and eliminate the need for expensive setup preparation during inference. Extensive experiments and user studies confirm that our proposed approach achieves state-of-the-art performance.