# SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models

arXiv: https://arxiv.org/abs/2405.13336 · HTML: https://mlsanity.com/paper/2405.13336/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2024.

## Built on by (most cited first, 18 of 18)

- [MOSPA: Human Motion Generation Driven by Spatial Audio](https://mlsanity.com/paper/2507.11949/) (2025)
- [Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models](https://mlsanity.com/paper/2507.20220/) (2025)
- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.22731/) (2025)
- [CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation](https://mlsanity.com/paper/2511.22863/) (2025)
- [fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment](https://mlsanity.com/paper/2512.01189/) (2025)
- [ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interaction](https://mlsanity.com/paper/2602.14048/) (2026)
- [ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis](https://mlsanity.com/paper/2503.06499/) (2025)
- [SARAH: Spatially Aware Real-time Agentic Humans](https://mlsanity.com/paper/2602.18432/) (2026)
- [EchoAvatar: Real-time Generative Avatar Animation from Audio Streams](https://mlsanity.com/paper/2605.28272/) (2026)
- [Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark](https://mlsanity.com/paper/2511.01233/) (2025)
- [Alignment-Process-Outcome: Rethinking How AIs and Humans Collaborate](https://mlsanity.com/paper/2603.08017/) (2026)
- [Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation](https://mlsanity.com/paper/2609.00369/) (2026)
- [MoSAT: Human Motion Generation from Spatial Audio and Textual Description](https://mlsanity.com/paper/2609.23797/) (2026)
- [Co-speech Gesture Video Generation via Motion-Based Graph Retrieval](https://mlsanity.com/paper/2512.02576/) (2025)
- [SiGnature: Explicit Motion Diffusion for Stylized Semantic Gesture](https://mlsanity.com/paper/2606.15889/) (2026)
- [LLM-Based Selection of Incongruent Verbal and Nonverbal Behavior for Virtual Humans](https://mlsanity.com/paper/2608.22731/) (2026)
- [Super Star: Towards Streaming Real-time Interactive Agents for Digital Humans](https://mlsanity.com/paper/2608.24909/) (2026)

## Abstract (arXiv)

The automated synthesis of high-quality 3D gestures from speech is of significant value in virtual humans and gaming. Previous methods focus on synthesizing gestures that are synchronized with speech rhythm, yet they frequently overlook the inclusion of semantic gestures. These are sparse and follow a long-tailed distribution across the gesture sequence, making them difficult to learn in an end-to-end manner. Moreover, generating gestures, rhythmically aligned with speech, faces a significant issue that cannot be generalized to in-the-wild speeches. To address these issues, we introduce SIGGesture, a novel diffusion-based approach for synthesizing realistic gestures that are of both high quality and semantically pertinent. Specifically, we firstly build a strong diffusion-based foundation model for rhythmical gesture synthesis by pre-training it on a collected large-scale dataset with pseudo labels. Secondly, we leverage the powerful generalization capabilities of Large Language Models (LLMs) to generate proper semantic gestures for the various speech content. Finally, we propose a semantic injection module to infuse semantic information into the synthesized results during diffusion reverse process. Extensive experiments demonstrate that the proposed SIGGesture significantly outperforms existing baselines and shows excellent generalization and controllability.