# Intentional Gesture: Deliver Your Intentions with Gestures for Speech

arXiv: https://arxiv.org/abs/2505.15197 · HTML: https://mlsanity.com/paper/2505.15197/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 59 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 30 of 49)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [MACE: Mass Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2403.06135/) (2024)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition](https://mlsanity.com/paper/2307.12493/) (2023)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances](https://mlsanity.com/paper/2410.18775/) (2024)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [EraseAnything: Enabling Concept Erasure in Rectified Flow Transformers](https://mlsanity.com/paper/2412.20413/) (2024)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation](https://mlsanity.com/paper/2401.04747/) (2024)

## Built on by (most cited first, 8 of 8)

- [MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness](https://mlsanity.com/paper/2505.20426/) (2025)
- [LiveGesture Streamable Co-Speech Gesture Generation Model](https://mlsanity.com/paper/2604.10927/) (2026)
- [SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching](https://mlsanity.com/paper/2603.26553/) (2026)
- [Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarks](https://mlsanity.com/paper/2511.05598/) (2025)
- [FADE: Adversarial Concept Erasure in Flow Models](https://mlsanity.com/paper/2507.12283/) (2025)
- [Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness](https://mlsanity.com/paper/2509.12024/) (2025)
- [TDMM-LM: Bridging Facial Understanding and Animation via Language Models](https://mlsanity.com/paper/2603.16936/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)

## Abstract (arXiv)

When humans speak, gestures help convey communicative intentions, such as adding emphasis or describing concepts. However, current co-speech gesture generation methods rely solely on superficial linguistic cues (e.g. speech audio or text transcripts), neglecting to understand and leverage the communicative intention that underpins human gestures. This results in outputs that are rhythmically synchronized with speech but are semantically shallow. To address this gap, we introduce Intentional-Gesture, a novel framework that casts gesture generation as an intention-reasoning task grounded in high-level communicative functions. First, we curate the InG dataset by augmenting BEAT-2 with gesture-intention annotations (i.e., text sentences summarizing intentions), which are automatically annotated using large vision-language models. Next, we introduce the Intentional Gesture Motion Tokenizer to leverage these intention annotations. It injects high-level communicative functions (e.g., intentions) into tokenized motion representations to enable intention-aware gesture synthesis that are both temporally aligned and semantically meaningful, achieving new state-of-the-art performance on the BEAT-2 benchmark. Our framework offers a modular foundation for expressive gesture generation in digital humans and embodied AI. Project Page: https://andypinxinliu.github.io/Intentional-Gesture